[SPARK][FLINK] job ownership facet

Problem
Spark and Flink integration should be capable of passing job owner information from config entries that should result in ownership facet being attached to OL event.
Solution

add config entry in java client

consume config entry in both flink & spark job facets generation code

refactor on Spark side to include OpenLineageYaml in OpenLineageContext. This should be useful for other features in future as well.

TODO: docs PR will be prepared once this PR gets approved.

Note: All schema changes require discussion. Please link the issue for context.

[ ] Your change modifies the core OpenLineage model
[ ] Your change modifies one or more OpenLineage facets

If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports S3 and GCS filesystem operations, tested with AWS EMR).
One-line summary:
Checklist

[ ] You've signed-off your work
[x] Your pull request title follows our guidelines
[x] Your changes are accompanied by tests (if relevant)
[ ] Your change contains a small diff and is self-contained
[ ] You've updated any relevant documentation (if relevant)
[ ] Your comment includes a one-liner for the changelog about the specific purpose of the change (if necessary)
[ ] You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
[ ] You've added a header to source files (if relevant)

SPDX-License-Identifier: Apache-2.0
Copyright 2018-2023 contributors to the OpenLineage project

@pawel-big-lebowski what if you have multiple different jobs with different owners using same cluster? I don't think there's a way to properly assign ownership using proposed mechanism then.
What if we added some jobName -> owners mapping in the config? Is getting proper job name too much of a hurdle for users?

@pawel-big-lebowski what if you have multiple different jobs with different owners using same cluster? I don't think there's a way to properly assign ownership using proposed mechanism then.
What if we added some jobName -> owners mapping in the config? Is getting proper job name too much of a hurdle for users?

This seems to be another problem. We do have two ways for providing config entries:

provide a file openlineage.yaml which is convenient to have global OL settings per all the job run in the cluster
Spark config entries which is convenient for per-job basis

However, we're missing some way to combine those two approaches, which would make totally sense like: take common config entries from openlineage.yaml and mix it with job specific spark conf. Otherwise we end up with two worlds of OpenLineage configuration entries, as we are now.

Here are my thoughts:

integration-specific configuration: settings unique to each integration (like Spark) should be defined solely within that integration's configuration system (e.g. Spark configuration). This keeps things clean and avoids overloading the client configuration.
common client-side configuration: general settings applicable across integrations are in the client configuration. This allows for centralized management and easier updates. You can leverage approaches like providing an openlineage.yaml file or environment variables for client configuration.
integration-to-client configuration translation to bridge the gap: if specific integration settings affect the client's behavior, there should be a mechanism to translate them into client-side configuration. This ensures the client receives the necessary information regardless of the initial configuration source. In Airflow you can use Airflow's configuration mechanism to configure Python client (unfortunately, we have to maintain old environment variables as well, that's a different story).

global vs. per-job:
While the above approach simplifies configuration, including global or per-job analysis settings might introduce unnecessary complexity. Maybe having global configuration per client (or having some default) and overriding this per job via integration config could solve the problem?
A bit of side-note: Python client should have similar way to provide Job ownership, shouldn't it?

Here are my thoughts:

integration-specific configuration: settings unique to each integration (like Spark) should be defined solely within that integration's configuration system (e.g. Spark configuration). This keeps things clean and avoids overloading the client configuration.

Job ownership facet is not such an example as it's common for Spark and Flink, which is a good reason to keep its config in Java client library.

common client-side configuration: general settings applicable across integrations are in the client configuration. This allows for centralized management and easier updates. You can leverage approaches like providing an openlineage.yaml file or environment variables for client configuration.

I really would like to avoid explaining users where to put each config entry. It makes sense for a user to put common environment related properties into openlineage.yaml while keeping job specific ones in spark.conf. But we shouldn't enforce any distinction of config properties on our end.
From Spark & Flink integration side, it would have been useful to have a single object containing all the config entries to be passed with the context. I think OpenLineageYaml is the object to do this. I think it should be extended to contain Map<String, String> to be contain other config entries which does not have to be used purely by java client.

integration-to-client configuration translation to bridge the gap: if specific integration settings affect the client's behavior, there should be a mechanism to translate them into client-side configuration. This ensures the client receives the necessary information regardless of the initial configuration source. In Airflow you can use Airflow's configuration mechanism to configure Python client (unfortunately, we have to maintain old environment variables as well, that's a different story).

I think that even if we have non-client-library config entries, we should be able to access them through client library classes (like OpenLineageYaml), because having a consistent way to pass configuration from user perspective is more important.

global vs. per-job:
While the above approach simplifies configuration, including global or per-job analysis settings might introduce unnecessary complexity. Maybe having global configuration per client (or having some default) and overriding this per job via integration config could solve the problem?
A bit of side-note: Python client should have similar way to provide Job ownership, shouldn't it?

It makes sense for a user to put common environment related properties into openlineage.yaml

Deploying openlineage.yml file in different Spark modes (client, cluster) may be a tricky part. IMHO it's better if configuration can be passed via either .yml file or via explicitly set Spark session config.

Job ownership facet is not such an example as it's common for Spark and Flink, which is a good reason to keep its config in Java client library.

I think pure facet is one thing, how we're using it can be a separate issue. There is no prescription that we can't have per-integration configuration that exists in two different integrations slightly differently, if that's the best solution.

I really would like to avoid explaining users where to put each config entry. It makes sense for a user to put common environment related properties into openlineage.yaml while keeping job specific ones in spark.conf. But we shouldn't enforce any distinction of config properties on our end.

I would avoid putting anything integration-specific to openlineage.yaml regardless on whether it's used per-job, per-cluster or in some other way.

From Spark & Flink integration side, it would have been useful to have a single object containing all the config entries to be passed with the context. I think OpenLineageYaml is the object to do this. I think it should be extended to contain Map<String, String> to be contain other config entries which does not have to be used purely by java client.

I very much disagree - if anything, OpenLineageYaml should be a source (or one of them) to create integration-specific object with relevant context. We should deal less in stringly comparisons, and more in constructed objects from that configuration. In Spark, this way of dealing with things already exists - OpenLineageContext.

integration-to-client configuration translation to bridge the gap: if specific integration settings affect the client's behavior, there should be a mechanism to translate them into client-side configuration. This ensures the client receives the necessary information regardless of the initial configuration source. In Airflow you can use Airflow's configuration mechanism to configure Python client (unfortunately, we have to maintain old environment variables as well, that's a different story).

I would go the opposite way - client-specific config should be translated to integration-specific one. Client - or rather what it's becoming - a generic helper library should provide mechanisms that integration should define how to use. For example, client provides circuit breaker or metrics - but it's up to integration to use it.

@mobuchowski To wrap up the thoughts to the context of this PR: your suggestion is to use only Spark & Flink conf entries for job ownership facet while not changing anything in openlineage-java. The leads to some duplicated code, but it's better because this feature does not have anything in common with openlineage-java. Users will not be able to configure job ownership over openlineage.yaml and this shouldn't be considered a problem.
Is my understanding correct?

@pawel-big-lebowski we can provide a generic mechanism in openlineage-java - but it can't assume how particular integrations will use it, especially that they will serialize some data to untyped map Map<String, String> that would be passed with OpenLineageYaml.
I think circuit breaker and metrics are good examples of desired behavior - openlineage-java acting as a library - but I'm not sure if it would be the best solution here, as it can't clearly deal with per-job vs per-cluster config, so I would consider mechanism specific to the integration here.

BTW, if we're discussing generic configuration, I would rather have OpenLineageConfig (changed from OpenLineageYaml) that would be derived by integration-specific ones (like SparkOpenLineageConfig) which would create internal object used in OpenLineageContext - there should be no need for stringly typed checks inside code outside of the objects created by it - it strongly couples general code to some configuration options, rather than to interfaces.

I see that changelog entry was added, but for 1.12.0 instead of Unreleased


Update log severity for Debug Statements in hooks.server.ts

The severity was initially set to console.log, which would lead to the UI logs being flooded with successfully sso'd statements when using an SSO config. I've altered these to console.debug.

@Mythicaeda sorry...it looks like debug is just an alias for log on node... If you think that it's too verbose, then we could probably just remove it since it seems like it's less necessary to know of the success state.

Oh darn... 😔 I'll remove the statements then.


Issues on visit & user id logs

Hi !
First of all, thank you for maintaining this package :)
I am quite new in matomo so I would like to apologize if missunderstood some concepts.
I have tried it in a very easy way by initializing the package and then logging some events or screens.
Events & Screens logging work very well but I don't understand how the visits count and the user ID work and I think their might have a bug:
Event if I open & close many times the app, all the events & the screens are logged in the same Visit with the same Visitor ID.
Correct me if I am wrong but, as I understood the package (and as other analytics package work):

if a user ID is not set by setVisitorUserId() then a different user ID should be generated each time (like "anonymous" users)
each time I open the app (if it was closed), it should be logged as a new visit in the dashboard.

The app is pretty easy (it is just a little app to test matomo) and has only some initialisation:
void main() async {
  await MatomoTracker.instance.initialize(
      siteId: 42,
      url: 'my_url',
    );

   runApp(...);
}

And then some events logging:
MatomoTracker.instance.trackEvent(
      eventCategory: category,
      action: action,
      eventName: name,
      eventValue: value,
    );

Thank you in advance for your help.

Ok, my bad, I didn't read the code and I think I understand a bit better.
As I can see in the code, we are saving in shared preferences the generated User ID from one "session" to an other to be able to reuse it.
Same thing with visits count that is saved in shared preferences.
I also had a look into the official API (https://developer.matomo.org/api-reference/tracking-api) and saw this:
If a visit is found in the last 30 minutes with your specified User ID, then the new action will be recorded to this existing visit.

So.... what I understand is that, even if we are logging a new visit count each time, the API will look at the last visit and, if it was less than 30 minutes ago... will log the new events in the same visit, even if we log a new visit count.
I have tested everything after 30 minutes and I can confirm that it works that way.
But I still don't understand why, to be honest.

What is the point of this 30 minutes rule ? If I log events every 25 minutes, I could have only ONE visit whereas I have maybe opened / closed the app hundred times, that is quite strange

Shouldn't we choose if we want the visitor ID to be saved in shared preferences ?
If a user is not logged in, we could assume that every app instance is actually lunched by anonymous user and not always the same

Actually, why are the visits count and user ID not handled by the API ?
Shouldn't the API be the only one responsible for this logic ?
What would happen if I have both an Android & an iOS app (native) and one package use this saving user id method and the other one don't ?
Once again, so sorry for calling it a "Bug", that was obviously a missunderstanding.
I also may be asking all my questions to the wrong person and do not hesitate to tell me if it is the case

Hi @ThomasEcalle,

What is the point of this 30 minutes rule ? If I log events every 25 minutes, I could have only ONE visit whereas I have maybe opened / closed the app hundred times, that is quite strange

Actually, why are the visits count and user ID not handled by the API ?

We are not affiliated to the Matomo development team so I cannot say for sure why this behavior is in place on the API side. On another note we are actually keeping the visit count locally in the shared preferences and sending it with the _idvc query parameter to the API, but it has been removed in Matomo 4 (We should probably remove the code related to this btw)
Seems like there is nothing more we can do on our end.

Shouldn't we choose if we want the visitor ID to be saved in shared preferences ?
If a user is not logged in, we could assume that every app instance is actually lunched by anonymous user and not always the same

This should be doable by modifying a bit the "cookieless" behavior already implemented. For now, you can only specify the cookieless property in the initialize method, but by allowing to change it on the fly I think you might be able to obtain the result you are looking for.
If it's something you'd like to see in this package I invite you to open a feature request with some details to understand exactly what is your need and how the package should be solving it.

The idea behind the 30min thing is that matomo was developed for web usage. Where in the app world we know when a user closes or leaves the app, in the web world there is no such concept as "session".
For the app usage it would be helpfull to actually be aware of app closes and initiate new visits. The tracking API knows a parameter for that: new_visit. To fix the behaviour we could always set the new_visit parameter for the first event that is dispatched. I'd be happy to come up with something.

Thanks for the insights @EPNW-Eric!
Yeah, the new_visit param seems to be the way to go, feel free to open a PR, otherwise I can also work on this 👍


[10.0.0-dev] Assets aren't versioned

[x] I've read the guidelines for Contributing to Roots Projects
[x] This request isn't a duplicate of an existing issue
[x] I've read the docs and followed them (if applicable)
[x] This is not a personal support request that should be posted on the Roots Discourse community

Description
Hello Roots team!
I've been using Sage 10 for a few months now on quite a large project and as it nears completion, I've noticed that assets don't seem to be versioned even though mix.version() is on and I am seeing the hashed query strings in the generated dist/mix-manifest.json file.
Unless I'm missing something obvious, I assume files should be versioned when using the Roots\asset() helper function?
Steps to reproduce

Reference an asset using the helper function (eg. asset('styles/app.css')->uri() or just asset('styles/app.css')
Check to see output from this function - it should contain the versioned querystring (eg. styles/app.css?id=d797e3949596db483406)

Expected behavior: I expect the hashed querystring ID to be present.
Actual behavior: No querystring is appended to the asset URI.
Reproduces how often: This happens all the time, whether in development or production mode (yarn start / yarn build / yarn build:production)
Versions

Mac OS 10.14.6
PHP 7.3.13
Node v13.12.0
Sage 10.0.0-alpha.1

Additional information
I know that Sage 10 is still in alpha and I have a modified version that uses Stylus + Tailwind CSS + PurgeCSS but I've been pretty careful with my changes and updates so I don't think anything I've done has affected the versioning.
Any pointers to things I can check or test would be appreciated :)

Is your manifest in assets.php set to use mix-manifest.json?

Ah, that was it! So simple... Thank you Brandon!
I was missing this update https://github.com/roots/sage/commit/41e4f6d2972a8e984d00b9e910f97384e05d51c5 and it looks like there are a few others that I will need to merge in later. It's the price of using an alpha release but so far Sage 10 has been really nice to use 👍


readGENEActiv lux data

In R function readGENEActiv we currently divide by 9 for LUX as that seemed to work for all test data.
However, this does not always seem to be correct. To do:

Investigate more closely why this does not seem to provide consistent output.
Improve LUX code to become more robust across files.

Acceleration columns and timestamps seem to be correct though, so not a major concern as lux is rarely used.

Now fixed with today's commit https://github.com/wadpac/GGIRread/commit/f5b1c62ecef78ef0a943bdc5aeb679c6bccefcd6
(forgot to tag this issue).


[Publisher] Parameters added using swagger editor are not visible under parameters heading in API resource page

Issue:
When parameters are added to the API using swagger editor, they are not visible under parameter heading in the API resource page of API Publisher.
Steps to recreate:

Add a parameter to an existing API resource using the swagger editor.
Check in the API resource page.

The parameters added from the swagger editor are shown in the DevPortal Test Console. Hence reducing the severity.

Ref property are not shown as well. https://github.com/wso2/product-apim/issues/5273

Closing since the issue is tracked from https://github.com/wso2/product-apim/issues/5273


mimxrt1050_evk_qspi: mcuboot serial recovery: image upload fails after updating to Zephyr v3.3.0

Describe the bug
After upgrading to Zephyr v3.3.0 the image uploading during serial recovery mode is not working anymore.
I was working well before when I was using Zephyr V3.1.0
To Reproduce
usb_cdc_acm_log_recovery.conf is customized a bit to enable LED:
CONFIG_LOG=y

# The build won't fit on the partition allocated for it without size
# optimizations.
CONFIG_SIZE_OPTIMIZATIONS=y

# Serial
CONFIG_SERIAL=y
CONFIG_UART_LINE_CTRL=y

# MCUBoot serial
CONFIG_MCUBOOT_SERIAL=y
CONFIG_BOOT_SERIAL_CDC_ACM=y

CONFIG_LOG_BACKEND_UART=y
CONFIG_LOG_BACKEND_RTT=n

CONFIG_MCUBOOT_INDICATION_LED=y

and usb_cdc_acm.overlay have been customized to:
/ {
    aliases {
        mcuboot-led0 = &green_led;
	mcuboot-button0 = &user_button;
    };

	chosen {
		zephyr,uart-mcumgr = &zephyr_udc0;
	};
};

&zephyr_udc0 {
	cdc_acm_uart0 {
		compatible = "zephyr,cdc-acm-uart";
	};
};

I have tried without "chosen" also.
Steps to reproduce the behavior:

west build -b mimxrt1050_evk_qspi zephyrproject/bootloader/mcuboot/boot/zephyr/ -d build-boot -- -DOVERLAY_CONFIG="usb_cdc_acm_log_recovery.conf" -DDTC_OVERLAY_FILE="usb_cdc_acm.overlay;app.overlay"
west flash -d build-boot/
Build an app to use for uploading:
west build -b mimxrt1050_evk_qspi -d build-boot-app zephyrproject/bootloader/mcuboot/samples/zephyr/hello-world/ -- DCONFIG_BOOTLOADER_MCUBOOT=y '-DCONFIG_MCUBOOT_SIGNATURE_KEY_FILE="zephyrproject/bootloader/mcuboot/root-rsa-2048.pem"'
sudo ~/go/bin/mcumgr --conntype serial --connstring "/dev/ttyACM1,baud=115200,mtu=512" image upload build-boot-app/zephyr/zephyr.signed.bin -l debug

Expected behavior
That the image uploading would succeed
Logs and console output
The output from debug console
*** Booting Zephyr OS build zephyr-v3.3.0 ***
I: Starting bootloader
I: Enter the serial recovery mode
I: Device configured
W: endpoint 0x1 already occupied
I: Erasing range 0x%jx:0x%jx
I: Writing at 0x0 until 0x129
I: RX: 0x0
I: TX
I: Writing at 0x129 until 0x27d
I: RX: 0x0
I: TX

The output from mcumgr
00000060  6f 4f 63 44 4c 46 32 42  72 61 54 4b 42 50 73 50  |oOcDLF2BraTKBPsP|
00000070  74 63 4b 39 76 70 64 58  2b 2b 58 53              |tcK9vpdX++XS|
DEBU[2023-04-26 08:52:40.665] Tx serial
00000000  0a                                                |.|
DEBU[2023-04-26 08:52:40.685] Tx serial
00000000  04 14                                             |..|
DEBU[2023-04-26 08:52:40.685] Tx serial
00000000  76 61 5a 4c                                       |vaZL|
DEBU[2023-04-26 08:52:40.685] Tx serial
00000000  0a                                                |.|
DEBU[2023-04-26 08:52:40.761] Rx serial:
00000000  06 09 41 41 6f 44 41 41  41 41 41 41 46 43 41 59  |..AAoDAAAAAAFCAY|
00000010  54 4b                                             |TK|
DEBU[2023-04-26 08:52:40.761] Decoded input:
00000000  03 00 00 00 00 01 42 01                           |......B.|
DEBU[2023-04-26 08:52:40.761] rx nmp response: 00000000  03 00 00 00 00 01 42 01                           |......B.|
DEBU[2023-04-26 08:52:40.761] Failure decoding NMP rsp: Invalid response: EOF

Environment (please complete the following information):

OS: Linux 22.04
Toolchain Zephyr 0.15.2
Zephyr v3.3.0

Can you give a try with https://github.com/zephyrproject-rtos/zephyr/pull/57256 ?

After some more investigation, it seems like the qspi is being flashing correctly, but the response received after each bulk is wrong - causing it to take a very long time to flash, and the progress bar from mcumgr also stays at 0%.

 During that time, the module is probably "dead" but the go application will resend packets if it does not get a response which might cause problems with them. I think someone else was reporting an issue with USB CDC packets but on an STM32 board, @Frank-Buss maybe?

A quick analysis based on logging timestamps, writing does not take a long time
Output from minicom, where "Writing at 0xX until 0xXXX" is just before entering write operations, and "RX" is after.
So less than 3ms per 340 bytes.
[2023-04-26 13:09:43.087] I: Writing at 0x0 until 0x129
[2023-04-26 13:09:43.090] I: RX: 0x0
[2023-04-26 13:09:43.090] I: TX
[2023-04-26 13:09:46.094] I: Writing at 0x129 until 0x27d
[2023-04-26 13:09:46.097] I: RX: 0x0
[2023-04-26 13:09:46.098] I: TX
[2023-04-26 13:09:49.179] I: Writing at 0x27d until 0x3d1
[2023-04-26 13:09:49.182] I: RX: 0x0
[2023-04-26 13:09:49.183] I: TX
[2023-04-26 13:09:52.264] I: Writing at 0x3d1 until 0x525
[2023-04-26 13:09:52.266] I: RX: 0x0
[2023-04-26 13:09:52.267] I: TX
[2023-04-26 13:09:55.349] I: Writing at 0x525 until 0x679
[2023-04-26 13:09:55.352] I: RX: 0x0
[2023-04-26 13:09:55.352] I: TX

But when only 340 bytes is sent every 3 seconds it's taking a long time - I will try using another mcumgr version

How long does it take to perform the QSPI erase/write? During that time, the module is probably "dead" but the go application will resend packets if it does not get a response which might cause problems with them. I think someone else was reporting an issue with USB CDC packets but on an STM32 board, @Frank-Buss maybe?

I found multiple problems. When erasing external flash, it needed too long and the timeout didn't work, because it is hardcoded to 3s. I fixed it:
https://github.com/apache/mynewt-newtmgr/pull/198
Not merged since a month, I guess the project is dead.
The Go program is horrible in other ways as well, which is the reason I started my own mcumgr-client program. I then encountered a problem with USB CDC in Zephyr with STM32 microcontrollers, see the notes in my program how to fix it:
https://github.com/vouch-opensource/mcumgr-client#notes
The underlying problem might be different, but I guess no point in investigating it more deeply, because the new Zephyr USB stack is around the corner.

Can you also retry using main of zephyr (remember to west update) and see if the problem is still present there as maybe it has been fixed?

Just tested with main and there is even more issues - mcuboot stops quite early in main without even initializing the usb.

Here is some debugging:
*** Booting Zephyr OS build zephyr-v3.3.0-3225-ga004a85ccf53 ***
[00:00:00.001,000] <dbg> os: k_sched_unlock: scheduler unlocked (0x800090a0:0)
[00:00:00.001,000] <dbg> mpu: mpu_configure_region: Configure MPU region at index 0x3
[00:00:00.001,000] <dbg> mpu: region_allocate_and_init: Program MPU region at index 0x3
[00:00:00.001,000] <dbg> mpu: region_init: [3] 0x80013440 0x150b000a
[00:00:00.001,000] <inf> mcuboot: Starting bootloader
[00:00:00.001,000] <dbg> os: setup_thread_stack: stack 0x80012300 for thread 0x80008ac0: obj_size=832 buf_s0
[00:00:00.001,000] <inf> mcuboot: Enter the serial recovery mode
[00:00:00.001,000] <dbg> usb_device: usb_enable: lock usb_enable_lock mutex
[00:00:00.001,000] <dbg> os: z_impl_k_mutex_lock: 0x800090a0 took mutex 0x800083c4, count: 1, orig prio: 0
[00:00:00.031,000] <dbg> mpu: mpu_configure_region: Configure MPU region at index 0x3
[00:00:00.031,000] <dbg> mpu: region_allocate_and_init: Program MPU region at index 0x3
[00:00:00.031,000] <dbg> mpu: region_init: [3] 0x80012300 0x150b000a
[00:00:00.228,000] <dbg> os: z_tick_sleep: thread 0x80008ac0 for 300 ticks
[00:00:00.228,000] <dbg> mpu: mpu_configure_region: Configure MPU region at index 0x3
[00:00:00.228,000] <dbg> mpu: region_allocate_and_init: Program MPU region at index 0x3
[00:00:00.228,000] <dbg> mpu: region_init: [3] 0x80013440 0x150b000a
[00:00:00.258,000] <dbg> mpu: mpu_configure_region: Configure MPU region at index 0x3
[00:00:00.258,000] <dbg> mpu: region_allocate_and_init: Program MPU region at index 0x3
[00:00:00.258,000] <dbg> mpu: region_init: [3] 0x80012300 0x150b000a
[00:00:00.312,000] <dbg> os: z_tick_sleep: thread 0x80008ac0 for 300 ticks
[00:00:00.312,000] <dbg> mpu: mpu_configure_region: Configure MPU region at index 0x3
[00:00:00.312,000] <dbg> mpu: region_allocate_and_init: Program MPU region at index 0x3

it is stuck in a loop or something, a minute later, it is still printing the same with a high frequency:
[00:01:46.098,000] <dbg> os: z_tick_sleep: thread 0x80008ac0 for 300 ticks
[00:01:46.098,000] <dbg> mpu: mpu_configure_region: Configure MPU region at index 0x3
[00:01:46.098,000] <dbg> mpu: region_allocate_and_init: Program MPU region at index 0x3
[00:01:46.098,000] <dbg> mpu: region_init: [3] 0x80013440 0x150b000a

Maybe this fixes the issue you are currently seeing: https://github.com/zephyrproject-rtos/zephyr/pull/57313

Seems that NXP has their own MPU file which is different to the normal ARM one. @mmahadevan108 @dlech seems like a bug in the NXP MPU code?

Could it be related to the newer updates with cbor? Since the issue seems to be found here in the mcumgr tool.
<EMAIL_ADDRESS>cborCodec := new(codec.CborHandle)
dec := codec.NewDecoderBytes(body, cborCodec)

if err := dec.Decode(r); err != nil {
return nil, fmt.Errorf("Invalid response: %s", err.Error())
}

Maybe mcuboot is not responding with a valid cbor message anymore?

If the output on the go application is correct then there is a missing 0x0a character from the end of the response. The response decodes to 00 0a 03 00 00 00 00 01 42 01 84 ca which is length 0x000a, OP 0x03 (write response), data length 0x0000, group: 0x0001 (application), sequence: 0x42, command: 0x01, crc: 0x84ca
So you are getting the correct response minus the expected newline character, maybe check with something like wireshark USB monitoring to see if you are really getting the final 0x0a character or not.

Just did a test monitoring usb traffic.
This is where the reponse is received by mcumgr
DEBU[2023-04-27 14:15:32.553] Tx serial
00000000  0a                                                |.|
DEBU[2023-04-27 14:15:32.555] Rx serial:
00000000  06 09 41 41 6f 44 41 41  41 41 41 41 47 34 41 58  |..AAoDAAAAAAG4AX|
00000010  6a 41                                             |jA|
DEBU[2023-04-27 14:15:32.555] Decoded input:
00000000  03 00 00 00 00 01 b8 01                           |........|
DEBU[2023-04-27 14:15:32.555] rx nmp response: 00000000  03 00 00 00 00 01 b8 01                           |........|
DEBU[2023-04-27 14:15:32.555] Failure decoding NMP rsp: Invalid response: EOF
packet=
00000000  03 00 00 00 00 01 b8 01                           |........|

And this is the data captured by a tcpdump:
060941416f44414141414141473441586a410a

It looks fine right? and with an ending newline '0a'

I just tried @Frank-Buss tool and it also throws an error while parsing the reponse.
13:36:50 [ERROR] Error: EOF while parsing a value

Sounds like a bug with your client @Frank-Buss ? Not really sure what to suggest, not had any of these issues even with the go application on nordic devices

I guess something must have been introduced since it was working fine with Zephyr v3.1.0. I will wait and see if any from NXP reacts on this issue :-)

@mmahadevan108 Any ideas here? I can reproduce the issue on an RT1060 EVK.
The USB CDC subsystem appears to write 19 bytes of data, and I do see 19 bytes transferred in a wireshark capture:
0000  06 09 41 41 6f 42 41 41  41 41 41 41 46 43 41 42   ··AAoBAA AAAAFCAB
0010  74 4e 0a                                           tN·

However, MCUMGR reports only receiving 18 bytes of data:
DEBU[2023-05-03 22:53:20.284] Rx serial:
00000000  06 09 41 41 6f 42 41 41  41 41 41 41 46 43 41 42  |..AAoBAAAAAAFCAB|
00000010  74 4e                                             |tN|
DEBU[2023-05-03 22:53:20.284] Decoded input:
00000000  01 00 00 00 00 01 42 00                           |......B.|
DEBU[2023-05-03 22:53:20.284] rx nmp response: 00000000  01 00 00 00 00 01 42 00

I cannot reproduce this issue on a K64 board, so I suspect the problem may be related to our USB driver.

Sounds like a bug with your client @Frank-Buss ? Not really sure what to suggest, not had any of these issues even with the go application on nordic devices

I don't think it is a bug in my tool, because the Go program reports the same EOF problem. It looks like the CBOR encoded part after the binary header is missing. Here is a full example what the Go program returns with the "image list" program (note, the final 0x0a is not listed as "Rx serial", but it is there, otherwise it would hang forever, which I had once as well, because of a bug in the STM32 USB driver) :
DEBU[2023-05-04 06:28:06.165] Using connection profile: name=acm0 type=serial connstring=dev=/dev/ttyACM0,baud=115200,mtu=512
DEBU[2023-05-04 06:28:06.166] {add-nmp-listener} [serial_sesn.go:213] seq=16
DEBU[2023-05-04 06:28:06.166] Encoded &{NmpBase:{hdr:{Op:0 Flags:0 Len:0 Group:1 Seq:16 Id:0}}} to:
00000000  a0                                                |.|
DEBU[2023-05-04 06:28:06.166] Encoded:
00000000  00 00 00 01 00 01 10 00  a0                       |.........|
DEBU[2023-05-04 06:28:06.166] Tx NMP request: 00000000  00 00 00 01 00 01 10 00  a0                       |.........|
DEBU[2023-05-04 06:28:06.166] Base64 encoding request:
00000000  00 00 00 01 00 01 10 00  a0                       |.........|
DEBU[2023-05-04 06:28:06.166] Tx serial
00000000  06 09                                             |..|
DEBU[2023-05-04 06:28:06.166] Tx serial
00000000  41 41 73 41 41 41 41 42  41 41 45 51 41 4b 44 46  |AAsAAAABAAEQAKDF|
00000010  6e 51 3d 3d                                       |nQ==|
DEBU[2023-05-04 06:28:06.166] Tx serial
00000000  0a                                                |.|
DEBU[2023-05-04 06:28:06.168] Rx serial:
00000000  06 09 41 42 55 42 41 41  41 4c 41 41 45 51 41 4c  |..ABUBAAALAAEQAL|
00000010  39 6d 61 57 31 68 5a 32  56 7a 6e 2f 2f 2f 57 46  |9maW1hZ2Vzn///WF|
00000020  59 3d                                             |Y=|
DEBU[2023-05-04 06:28:06.168] Decoded input:
00000000  01 00 00 0b 00 01 10 00  bf 66 69 6d 61 67 65 73  |.........fimages|
00000010  9f ff ff                                          |...|
DEBU[2023-05-04 06:28:06.168] rx nmp response: 00000000  01 00 00 0b 00 01 10 00  bf 66 69 6d 61 67 65 73  |.........fimages|
00000010  9f ff ff                                          |...|
DEBU[2023-05-04 06:28:06.168] Received nmp rsp: &{NmpBase:{hdr:{Op:1 Flags:0 Len:11 Group:1 Seq:16 Id:0}} Rc:0 Images:[] SplitStatus:N/A}
DEBU[2023-05-04 06:28:06.168] {remove-nmp-listener} [serial_sesn.go:213] seq=16
Images:
Split status: N/A (0)

PS: my tool has also a "-v" switch for verbose output. Looks like this:
frank@hal9000:~$ mcumgr-client -v -d /dev/ttyACM0 list
mcumgr-client 0.0.1, Copyright © 2023 Vouch.io LLC

04:49:29 [INFO] send image list request
04:49:29 [DEBUG] (1) mcumgr_client::transfer: request header: NmpHdr { op: Read, flags: 0, len: 1, group: Image, seq: 72, id: 0 }
04:49:29 [DEBUG] (1) mcumgr_client::transfer: serialized:<PHONE_NUMBER>014800a0
04:49:29 [DEBUG] (1) mcumgr_client::transfer: encoded with packet length and checksum: 000b0000000100014800a032f2
04:49:29 [DEBUG] (1) mcumgr_client::transfer: encoded: AAsAAAABAAFIAKAy8g==
04:49:29 [DEBUG] (1) mcumgr_client::transfer: result string: ABUBAAALAAFIAL9maW1hZ2Vzn///H2g=
04:49:29 [DEBUG] (1) mcumgr_client::transfer: response header: NmpHdr { op: ReadRsp, flags: 0, len: 11, group: Image, seq: 72, id: 0 }
04:49:29 [DEBUG] (1) mcumgr_client::transfer: cbor: bf66696d616765739fffff
04:49:29 [INFO] response: {
  "images": []
}

But would be nice to also show the transmitted and received bytes, I added an issue for it, should be easy to add:
https://github.com/vouch-opensource/mcumgr-client/issues/5

Some more info about this failure: I can get the SMR_SVR sample samples/subsys/mgmt/mcumgr/smp_svr to work correctly with the RT1060 over CDC ACM, but MCUBoot with mcumgr fails. In the working sample, I see the ACM driver class sends 202 bytes as a response. For the failing case only 19 bytes are sent. This makes me suspect the RX side may have an issue. However, the data sent by mcumgr and the data received by MCUBoot seem to be the same:
host side (RT1060 target):
DEBU[2023-05-04 10:49:09.832] Tx serial
00000000  06 09                                             |..|
DEBU[2023-05-04 10:49:09.832] Tx serial
00000000  41 41 73 41 41 41 41 42  41 41 46 43 41 4b 44 31  |AAsAAAABAAFCAKD1|
00000010  4d 77 3d 3d                                       |Mw==|
DEBU[2023-05-04 10:49:09.832] Tx serial
00000000  0a

device side (RT1060):
D: dev 0x6000ebe0 size 1 rx_ringbuf space 1024
RX: 06 09 41 41 73 41 41 41 41 42 41 41 46 43 41 4B 44 31 4D 77 3D 3D 0A
D: Got 19 bytes from ringbuffer send to ep 82

On the K64, the initial data received is similar:
host side (K64 target):
DEBU[2023-05-04 12:09:42.907] Tx serial
00000000  06 09                                             |..|
DEBU[2023-05-04 12:09:42.907] Tx serial
00000000  41 41 73 41 41 41 41 42  41 41 46 43 41 4b 44 31  |AAsAAAABAAFCAKD1|
00000010  4d 77 3d 3d                                       |Mw==|
DEBU[2023-05-04 12:09:42.907] Tx serial
00000000  0a

device side (K64):
D: dev 0xb664 size 1 rx_ringbuf space 1024
RX: 06 09 41 41 73 41 41 41 41 42 41 41 46 43 41 4B 44 31 4D 77 3D 3D 0A
D: Got 202 bytes from ringbuffer send to ep 83

@nordicjm Any insight as to why MCUBoot might respond differently on the RT1060 versus the K64?

but MCUBoot with mcumgr fails

I assume by this you mean mcuboot with serial recovery, and not mcuboot with mcumgr enabled, because the latter would be bad.

@nordicjm Any insight as to why MCUBoot might respond differently on the RT1060 versus the K64?

I would assume a driver bug, there's no issue on nordic boards, and I don't remember seeing issues on stm32 when I last tested (which was some months ago)

@nordicjm I've looked at this some more, and don't believe it is a driver bug- it seems like there is a bug in MCUBoot.
After some debugging, it appears that the ZCBOR structure set up by the MCUBoot is invalid. Specifically, this call to zcbor_new_encode_state: https://github.com/zephyrproject-rtos/mcuboot/blob/2878eb4e32fabc14bc391c3221cbf285064d9db4/boot/boot_serial/src/boot_serial.c#L135:
zcbor_new_encode_state(cbor_state, 2, (uint8_t *)bs_obuf, (size_t)bs_obuf + sizeof(bs_obuf), 0);

Appears to be using the ZCBOR API incorrectly. It seems that the API call should be:
zcbor_new_encode_state(cbor_state, 2, (uint8_t *)bs_obuf, sizeof(bs_obuf), 0);

as the 4th parameter is expected to be the payload length: https://github.com/zephyrproject-rtos/zcbor/blob/67fd8bb88d3136738661fa8bb5f9989103f4599e/include/zcbor_encode.h#L272
When debugging on the RT1050, bs_obuf + sizeof(bsobuf) comes out to be 0x800114d. The pointer bs_obuf is at 0x8001143.
The implementation of zcbor_new_encode_state does the following:
state_array[0].payload = payload;
state_array[0].payload_end = payload + payload_len;
state_array[0].elem_count = elem_count;

So for a platform like the RT1050 where the SRAM address space is at a high address, payload + payload_len will cause an integer overflow. On the FRDM-K64, the SRAM range is at 0x2000_0000, so roughly doubling the address like occurs here will not cause an overflow.
We can see the result of this, where payload_end comes out to be 0x2291 on the RT1050. This becomes an issue when constructing a ZCBOR object, as checks like the following will now fail:
#define ZCBOR_CHECK_PAYLOAD() \
	ZCBOR_ERR_IF(state->payload >= state->payload_end, ZCBOR_ERR_NO_PAYLOAD)

This results in the RT1050 responding with invalid data to the request from mcumgr, as the ZCBOR structure itself used to encode image data is invalid, and therefore never has image info stored within it.


Set config-controller version label on CM App CRs

Towards https://github.com/giantswarm/giantswarm/issues/14993

Back to draft. I need to rethink couple of things and add tests.

Back to draft. I need to rethink couple of things and add tests.

Now it's ready.

Now it's ready.


gatsby-source-prismic - invalid plugin options || Expected a value of type record<string,object> for schemas but received undefined.

Hello
Sorry, I've been scratching my head on this one for a while. It was building fine about a week ago, maybe 10 days, but now it just won't build.
I'm running on 3.0.0.
gatsby-config.js
    {
      resolve: `gatsby-source-prismic`,
      options: {
        repositoryName: `formula1gym`,
        accessToken: `${process.env.API_KEY}`,
        //linkResolver: ({ node, key, value }) => post => `/${post.uid}`,
      },
    },

When I run gatsby clean && gatsby develop ...
info Deleting .cache, public
info Successfully deleted directories
success open and validate gatsby-configs - 0.051s
success load plugins - 0.758s
success onPreInit - 0.005s
success initialize cache - 0.009s
success copy gatsby files - 0.085s
success onPreBootstrap - 0.019s
success createSchemaCustomization - 0.007s

 ERROR

gatsby-source-prismic - invalid plugin options

 ERROR

Expected a value of type `record<string,object>` for `schemas` but received `undefined`.

  Error:

  - index.cjs:650 new StructError
    [formula1gym]/[superstruct]/lib/index.cjs:650:19

  - index.cjs:707 Function.Struct.assert
    [formula1gym]/[superstruct]/lib/index.cjs:707:13

  - index.cjs:679 Struct
    [formula1gym]/[superstruct]/lib/index.cjs:679:19

  - gatsby-node.js:73 validatePluginOptions
    [formula1gym]/[gatsby-source-prismic]/dist/gatsby-node.js:73:50

  - gatsby-node.js:1041
    [formula1gym]/[gatsby-source-prismic]/dist/gatsby-node.js:1041:25

  - Generator.next

  - gatsby-node.js:51
    [formula1gym]/[gatsby-source-prismic]/dist/gatsby-node.js:51:71

  - new Promise

  - gatsby-node.js:47 __awaiter
    [formula1gym]/[gatsby-source-prismic]/dist/gatsby-node.js:47:12

  - gatsby-node.js:1028 Object.sourceNodes
    [formula1gym]/[gatsby-source-prismic]/dist/gatsby-node.js:1028:55

  - api-runner-node.js:252 runAPI
    [formula1gym]/[gatsby]/dist/utils/api-runner-node.js:252:37

  - api-runner-node.js:371 resolve
    [formula1gym]/[gatsby]/dist/utils/api-runner-node.js:371:15

  - debuggability.js:384 Promise._execute
    [formula1gym]/[bluebird]/js/release/debuggability.js:384:9

  - promise.js:518 Promise._resolveFromExecutor
    [formula1gym]/[bluebird]/js/release/promise.js:518:18

  - promise.js:103 new Promise
    [formula1gym]/[bluebird]/js/release/promise.js:103:10

  - api-runner-node.js:370 Promise.mapSeries.plugin
    [formula1gym]/[gatsby]/dist/utils/api-runner-node.js:370:12

not finished source and transform nodes - 0.156s
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>dev: `gatsby clean && gatsby develop`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>dev script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR!      C:\Users\dan\AppData\Roaming\npm-cache\_logs\2020-03-14T12_18_44_481Z-debug.log

Anyone got any ideas?

Oh I've just seen this...
https://github.com/angeloashmore/gatsby-source-prismic/blob/master/docs/migrating-from-v2-to-v3.md
Will report back once I've read that!

Upgraded to v3.0.0, managed to get it working & building.

Glad you found the migration guide and got it working. 👍

@Zellement Since you've just gone through the process of upgrading from v2 to v3, I'm curious if you have any comments that might make the upgrade easier for others. Is there anything in the migration guide that could be worded better?
Thanks!

@angeloashmore Hey! It was OK - although I have run into a snag with images - it's a bit new to me, so they're displaying odd. I have some changes the client wants to do, so I'll be doing that soon - will report back ASAP.


⚠ PrivateHD has degraded performance

In f5b9110, PrivateHD ($PHD) experienced degraded performance:

HTTP code: 200
Response time: 2387 ms

Resolved: PrivateHD performance has improved in 3c23c5a.


pull request for e2e test at 11/15/2016 00:56:11

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

optest/docs/updated_at.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


[openrr-apps] Warn TOML config that is not valid as TOML

output example:
2022-01-20T04:15:25.912376Z  WARN openrr_apps::utils: config ./openrr-apps/config/sample_robot_client_config_for_urdf_viz.toml is not valid TOML: unexpected character found: `$` at line 44 column 1

+1


get_historical_klines ignores end_str

Describe the bug
Commit 50a852f introduced a bug in binance/client.py line 873, where it overrides the end_str parameter in get_historical_klines. This makes it impossible to manually set the end time.
To Reproduce
Code snippet to reproduce the behavior:
import datetime

start = datetime.datetime.now() - datetime.timedelta(hours=48)
end = datetime.datetime.now() - datetime.timedelta(hours=24)
data = client.get_historical_klines(ticker, interval, str(start), str(end))

Expected behavior
A clear and concise description of what you expected to happen.
data should only contain the data between start and end time.
Environment (please complete the following information):

Python version:  3.9
Virtual Env: pipenv
OS: Arch Linux
python-binance 0.7.11

I just noticed this bug as well. Came here to report it and found this issue.
The fix should probably be replacing the following line (client.py:873) from:
return self._historical_klines(symbol, interval, start_str, end_str=None, limit=500, spot=True)
to:
return self._historical_klines(symbol, interval, start_str, end_str, limit, spot)

I thought I was going mad, to the point I started playing with PostMan to confirm whether I was going mad or not. Then I found this issue.
+1

This is resolved in the feature/asyncio branch which I'll merge to master shortly. This will become v1.0.0


LSTM doesn't work with Symbols and HybridBlocks

I want to use LSTM with AWS SageMaker, but I am having problems with Symbols. I noticed that LSTM seems to make not attempt at using the HybridBlock pattern. For example, it only defines a forward method that inherently can't be used with Symbols,
    def forward(self, inputs, states=None):
        batch_size = inputs.shape[self._layout.find('N')]
        skip_states = states is None
        if skip_states:

Before I go about trying to fix this, I wanted to check if there were known issues, plans to fix it, or existing work done to fix this. Has any effort been made on this?

I have a question about its performance. The documentation says:

Recurrent cells allows fine-grained control when defining recurrent models. User can explicit step and unroll to construct complex networks. It provides more flexibility but is slower than recurrent layers.

So LSTMCell is slower than LSTM, but I'm wondering if hybridized then it's faster?

Our LSTM cell doesn't support full sequence hybridization yet. The hybridization happens only at cell level. Thus, it will still have performance penalty.

OK, thanks!


Update to apollo client 0.4.0

A few hacks we should fixe before merging

@stubailo thank you so much for doing this!

@jbaxleyiii I think it's ready to merge as soon as you have time to look it over!


fix: file-button testcodes fix

Closes #2374
Description

Current behavior (updates)

New behavior

Is this a breaking change (Yes/No):

Additional Information

@bearone236
We would appreciate it if you could correct the CI error.

@bearone236
We would appreciate it if you could correct the CI error.

Since it is a Vercel error, only Mr. Yamada himself would know the cause.
And I think it's because I'm creating a pull request from a forked repository.


Bump the version of Jackson

We're not going to get Jenkins feedback on this due to being on an older branch from before we had Jenkinsfiles, but I've run the unit/integration tests locally to validate.


Remove ossl_rsa_pss_params_30_set_maskgenalg().

This is an unused internal function. It should be added back
in if there is a need to set this value to something other than
NID_mgf1 (which is not likely).

Checklist

[ ] documentation is added or updated
[ ] tests are added or updated

This pull request is ready to merge

Merged.


Adds install targets for static libaries.

Also makes decNumberStatic.a be decNumber_static.a to be consistent with
ion_static.a
Previously, we were building these static library targets but not actually installing them.
Tested this by building/installing in a local prefix:
$ (mkdir -p build/dist && \
    cd build && cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$(pwd)/dist .. && \
    cmake --build . --target install -- -j30)
...
$ tree build/dist/lib
build/dist/lib
├── cmake
│   └── IonC
│       ├── IonCConfig.cmake
│       ├── IonCConfigVersion.cmake
│       ├── IonCTargets.cmake
│       └── IonCTargets-release.cmake
├── libdecNumber.so
├── libdecNumber_static.a
├── libgmock.a
├── libgmock_main.a
├── libgtest.a
├── libgtest_main.a
├── libionc.so -> libionc.so.1.0.3
├── libionc.so.1.0.3
└── libionc_static.a

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

LGTM. ion-cli will need a patch to accommodate the rename of decNumberStatic.

Interesting, let me take a look, I would've expected a build failure for this.

Taking a look at the CLI CMakeLists.txt:

The ion executable links against the shared libraries.
The ion_cli shared library links against the shared libraries.
The ion_cli_main archive appears to link against the shared libraries, which doesn't make any sense.  I think that particular CMake configuration is not doing anything useful here:

$ ar -t build/tools/cli/libion_cli_main.a
main.cpp.o
argtable3.c.o
cli.cpp.o

I just realized you meant amzn/ion-cli. Not the Ion CLI in this repository.


Introduce Multiversion support

Refactor the project to add multiversion support.
Material for discussion:

Kotlin instead of Java
which versions should be supported?

[ ] 1.8.8- 1.14
[ ] 1.12.2-1.14

All the versions :)

The default Minecraft worldborder is not available below 1.8 so all the versions is not possible.

Added some versions in the latest fork, I can think of simpler ways to do it (maybe we reflect). But for now it's better than no support. 🤔

Tested on 1.8.8 and 1.13 as well (ill try more later).

Reflection is a no go for that project. I will introduce a better abstraction layer some day.


Misspelling in README

"sensetive" is spelled incorrectly

Fixed by PR #1 (not yet pulled).

Thanks all , It merged now.


Datadog: update packages

Motivation for this change
Update datadog packages:

datadog agent to 6.11.0;
datadog process agent to 6.11.0;

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Assured whether relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

@GrahamcOfBorg build datadog-agent datadog-process-agent


successful job is re-enqueued during shutdown

It is possible that a job gets (falsely) re-enqueued during shutdow of a worker even though the job finished successfully.
Consider this example:
worker.end(true)
worker.join(5000)

So if the job does not throw an InterruptedException and takes less than 5 seconds, removeInFlight will re-enqueue the job as the worker has been set to SHUTDOWN_IMMEDIATE.
One solution would be to make removeInFlight aware of any exceptions. What do you think?

thanks for merging #147


Conditionally run composer:drupal-scaffold.

Only run the composer:drupal-scaffold task if it is defined in the
composer.json scripts section.

This fixes an issue where D7 builds (or D8  builds without the drupal-scaffold script defined) would fail with:
  [Symfony\Component\Console\Exception\CommandNotFoundException]
  Command "drupal-scaffold" is not defined.

Warning: Task "composer:drupal-scaffold" failed. Use --force to continue.

Aborted due to warnings.

Looks good, thanks!


[ansible-dojot] Alter context from v1beta1 to v1

I'm submitting a ...

[ ] bug report
[ ] feature request
[ ] support request

Do you want to request a feature or report a bug?

What is the current behavior?
Because the version of kubernetes up then we need change the clients who access his apis. So, the context name need to be chenged.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

What is the expected behavior?

What is the motivation / use case for changing the behavior?

Please tell us about your environment:

Version: 0.2.x
Environment: [kubernetes, docker-compose, standalone]
Operating system: [Ubuntu 16.04, Fedora 28, Ubuntu on VirtualBox (Windows 10), etc.]

Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, gitter, etc)

Fixed in v0.5.0-rc.1 by the #1770.


Trying to encode to FLAC with MediaFoundationEncoder but no success updated #2

Hello,
I am trying to encode an Audio-PCM Format to FLAC using the MediaFoundationEncoder.
Calling the MediaFoundationApi.EnumerateTransforms(MediaFoundationTransformCategories.AudioEncoder); tells me that the FLAC encoder should be available on my system and is defined like this.
Name: Microsoft FLAC Audio Encoder MFT
Input Types: 1 items:
    Audio-PCM
Class identifier: 128509e9-c44e-45dc-95e9-c255b8f466a6
Output Types: 1 items:
    Audio-0000f1ac-0000-0010-8000-00aa00389b71
Transform Flags: 1
Transform Category: Audio Encoder

If I understand this correctly it means that this encoder accepts an Audio-PCM input and supports the MFAudioFormat_FLAC output type represendet with the GUID 0000-0010-8000-00aa00389b71 .
Regarding to the documentation about Encode to MP3, WMA and AAC with MediaFoundationEncoder and Resampling Audio I created a short example to test this.
I read the _input.flac file with the MediaFoundationReader and resample it using the MediaFoundationResampler and write it back to my test folder using WaveFileWriter. This is working great.
However when I try to read the test file that I created like described in the  Encode to MP3, WMA and AAC with MediaFoundationEncoder I always get an error.
To make sure I did nothing wrong I read the WaveFormat of the existing FLAC file first and use it to create my MediaType for the MediaFoundationEncoder .
But when I call the enc.Encode(outputFlac, reader); methode I get this error:
The data specified for the media type is invalid, inconsistent, or not supported by this object.
How do I know what is wrong? The input media type should be ok because I am able to create an MP3 file without any issues.
Is there a specific format I have to use for the FLAC encoder? There is unfortunately no documentation about the FLAC encoder MFT.
Update
After digging more through the code in this repo I managed to find the part how I am able to list all supported values for a specific GUID.
Could it be possible that I used the decoder GUID for encoding?
After creating the GUID with new Guid("0000f1ac-0000-0010-8000-00aa00389b71") I am able to create FLAC files. The funny part here is that the encoding only works with 32Bit Audio-PCM formats.
Regarding to the FLAC documentation the input must be between 4 and 24 bits per sample.  No idea why it is working...
var availableMediaTypes = GetOutputMediaTypes(new Guid("0000f1ac-0000-0010-8000-00aa00389b71"))
                        .Where(mt => mt.SampleRate == 48000 && mt.ChannelCount == 1)
                        .Select(mt => new { MediaType = mt }) //, Delta = Math.Abs(desiredBitRate - mt.AverageBytesPerSecond * 8) })                                                              ////.OrderBy(mt => mt.Delta)
                        .OrderByDescending(mt => mt.MediaType.BitsPerSample)
                        .Select(mt => mt.MediaType);

Update 2
I managed now to make it work. Seems like I used the wrong codec so it was pure luck that it was working with 32bit Audio-PCM. 20 seconds where about 10 seconds in the output flac file.
Here's the working code sample that is working:
using (var reader = new MediaFoundationReader(testFilePath))
            {
                var availableMediaTypes = GetOutputMediaTypes(new Guid("0000f1ac-0000-0010-8000-00aa00389b71"))
                        .Where(mt => mt.SampleRate == reader.WaveFormat.SampleRate && mt.ChannelCount == reader.WaveFormat.Channels && mt.MediaType.BitsPerSample == reader.WaveFormat.BitsPerSample)
                        .Select(mt => new { MediaType = mt })
                        .OrderByDescending(mt => mt.MediaType.BitsPerSample)
                        .Select(mt => mt.MediaType);

                var mediaType = availableMediaTypes.FirstOrDefault();
                if (mediaType == null) throw new InvalidOperationException("No suitable FLAC encoders available");

                var outputFlac = Path.Combine(outputFolder, "baz.flac");

                using (var encoder = new MediaFoundationEncoder(mediaType))
                {
                    encoder.Encode(outputFlac, reader);
                }
            }

Maybe this could help someone in the near future who is facing the same problem.
Here's the code example I created:
var flacInputPath = Path.Combine(outputFolder, "_input.flac");

using (var reader = new MediaFoundationReader(flacInputPath))
{
	var outFormat = new WaveFormat(44100, reader.WaveFormat.Channels);

	using (var resampler = new MediaFoundationResampler(reader, outFormat))
	{
		WaveFileWriter.CreateWaveFile(outputWaveFilePath, resampler);
	}
}

using (var flacReader = new MediaFoundationReader(flacInputPath))
{
	using (var reader = new MediaFoundationReader(testFilePath))
	{
		var outputFlac = Path.Combine(outputFolder, "baz.flac");
		var mediaType = new MediaType(flacReader.WaveFormat);
		MediaFoundationEncoder enc = new MediaFoundationEncoder(mediaType);
		enc.Encode(outputFlac, reader);
	}
}

Thank you in advance!

@datoml Would you mind sharing the code you used to read the Transform properties?

@datoml Would you mind sharing the code you used to read the Transform properties?

I searched again the implementation and it seems that I had made the export in CPP and then call it using a "wrapper" in C#.
But I don't know exactly why I did it that way.


Use RHEL 9 as base image

... as soon as it's released.
One issue I found with AlmaLinux 9 is that libtinfo and ncurses-libs are only available in version 6 rather than 5. Maybe RHEL 9 is too new for the Source Engine?

Update: it's possible. POC branch: rhel-9
Basically we need to install ncurses-compat-libs.i686 into an almalinux:8.6 image and copy the libncurses and libtinfo libraries from there. After a bit of relinking and eventually performing ldconfig, the installation is complete and SRCDS doesn't complain about the libraries anymore. However, I think this is very hacky and I'm not sure how stable and reliable the servers are going to be in the long run.


Errors building from source (VS2015)

Apologies if this already exists; I can't find it :(
I've cloned the four repos that seem to be referenced from the various sln files in the Bridge repo (Bridge, Testing, Frameworks, Builder) and I've tried compiling them all; but none seem to have product a new bridge.js file that appears to have all the latest committed changes (and the .Testing one gave some compile errors).
Is it possible to document how to build from source?

http://bridge.net/kb/build-from-source/
bridge.js is generated by Bridge.csproj and can be located at ..\Bridge\Bridge\Resources
Can you please specify the compile errors in the .Testing.

Doh! Thanks :) I looked under the "Contributing" section in the KB, didn't spot that. Thanks :)
I'm building in VS 2015 RC which may be interfering with this, but the errors I get building .Testing are:
exception level: 0 - Index was outside the bounds of the array.	typeScriptBridge	C:\Work\Source\Bridge\Testing\tests\typeScriptBridge\EXEC

Missing compiler required member 'System.String.Format'	client	C:\Work\Source\Bridge\Testing\tests\client\CSC

The command ""C:\Work\Source\Bridge\Testing\tests\typeScriptBridge\..\..\..\Builder\Builder\bin\Debug\Bridge.Builder.exe" -p "C:\Work\Source\Bridge\Testing\tests\typeScriptBridge\typeScriptBridge.csproj" -b "C:\Work\Source\Bridge\Testing\tests\typeScriptBridge\bin\Debug\Bridge.dll"" exited with code -1.	typeScriptBridge

Btw, from the KB:

At this point, the full Bridge Solution should compile without errors, and fresh .dll’s will be available from the projects /bin directory.

It's not clear which directory this means. There are lots of bin/Debug folders with bridge.dll, but not the .js file. Should these be taken from different locations? Is the code that packages the NuGet file available in the repo (this would be the easiest way for us to reference the self-built version?)

I tried to replace the Bridge.dll/Bridge.html5.dll from the NuGet I'm referencing (a birty dirty, I know), but it then failed to compile with:
1>C:\Work\Source\DMS\ScrabbleTests\Bridge\packages\Bridge.1.7.0\build\Bridge.targets(12,5): error : Error: Index was outside the bounds of the array.    at Mono.Cecil.Mixin.ParseRuntime(String self)
1>C:\Work\Source\DMS\ScrabbleTests\Bridge\packages\Bridge.1.7.0\build\Bridge.targets(12,5): error :    at Mono.Cecil.PE.ImageReader.ReadMetadata()
1>C:\Work\Source\DMS\ScrabbleTests\Bridge\packages\Bridge.1.7.0\build\Bridge.targets(12,5): error :    at Mono.Cecil.PE.ImageReader.ReadImage()
1>C:\Work\Source\DMS\ScrabbleTests\Bridge\packages\Bridge.1.7.0\build\Bridge.targets(12,5): error :    at Mono.Cecil.PE.ImageReader.ReadImageFrom(Stream stream)
1>C:\Work\Source\DMS\ScrabbleTests\Bridge\packages\Bridge.1.7.0\build\Bridge.targets(12,5): error :    at Mono.Cecil.ModuleDefinition.ReadModule(Stream stream, ReaderParameters parameters)
1>C:\Work\Source\DMS\ScrabbleTests\Bridge\packages\Bridge.1.7.0\build\Bridge.targets(12,5): error :    at Mono.Cecil.ModuleDefinition.ReadModule(String fileName, ReaderParameters parameters)
1>C:\Work\Source\DMS\ScrabbleTests\Bridge\packages\Bridge.1.7.0\build\Bridge.targets(12,5): error :    at Bridge.Translator.Translator.LoadAssembly(String location, List`1 references)
1>C:\Work\Source\DMS\ScrabbleTests\Bridge\packages\Bridge.1.7.0\build\Bridge.targets(12,5): error :    at Bridge.Translator.Translator.InspectReferences()
1>C:\Work\Source\DMS\ScrabbleTests\Bridge\packages\Bridge.1.7.0\build\Bridge.targets(12,5): error :    at Bridge.Translator.Translator.Translate()
1>C:\Work\Source\DMS\ScrabbleTests\Bridge\packages\Bridge.1.7.0\build\Bridge.targets(12,5): error :    at Bridge.Build.GenerateScript.Execute()

My guess is that the tools\Bridge.Builder.dll is out of sync with the other DLLs. However, it doesn't look like a new Bridge.Builder.dll was generated as part of the build.
I think it would valuable to include some additional info on the build-from-source page about how best to use the build (the ideal would definitely be a script to build a new nupkg that's the same format as the one on nuget.org).

There are lots of bin/Debug folders with bridge.dll, but not the .js file.

bridge.js can be referenced from ..\Bridge\Bridge\Resources\bridge.js It is just for your reference as bridge.js is embedded into Bridge.dll Assemblies are referenced from corresponding projects. I.e. Bridge.dll from ..\Bridge\Bridge\bin; Bridge.Html5.dll from ..\Bridge\Html5\bin; jQuery from Frameworks\jQuery2\bin.

Is the code that packages the NuGet file available in the repo (this would be the easiest way for us to reference the self-built version?
My guess is that the tools\Bridge.Builder.dll is out of sync with the other DLLs. However, it doesn't look like a new Bridge.Builder.dll was generated as part of the build.

No, the code is not available. At least at the moment...
You can look at the the testing project https://github.com/bridgedotnet/Testing/blob/master/tests/client/client.csproj. Please note the PostBuildEvent how it references Bridge.Builder.exe and Bridge.dll (a custom way of using Bridge without NuGet packages).

I'm building in VS 2015 RC which may be interfering with this, but the errors I get building.

Yes, this appears to be VS 2015 related issues; we will have a look.

I think it would valuable to include some additional info on the build-from-source page about how best to use the build (the ideal would definitely be a script to build a new nupkg that's the same format as the one on nuget.org).

Agree)

@LeonidVeriga Thanks for the info; I'll give it a go!
Worth raising a case requesting a script to build the nupkg so it doesn't get lost?

We have an internal project for building the NuGet packages, and this is high on my list of items to refactor. While what we have works well, there is a better process that we need to move to.
I will expedite this NuGet package project.
Created Issue #323 to track this work.

@geoffreymcgill Great, thanks!

Hello,
I am currently using the script below (cygwin/linux/os x) when I want to overwrite the .DLLs of a NuGet package (attached on a project) with the latest, freshly built Bridge binaries:
#!/bin/bash

pjhome="${HOME}/Object.NET/projects/BridgeSolution1"
if [ ! -z "${1}" ]; then
 pjhome="$(realpath ${1})"
fi

brdhome="${pjhome}/packages/Bridge.1.7.0"
cd ${HOME}/Object.NET/Bridge.NET/git
#Bridge-main
cd Bridge/Html5/bin/Debug || { echo "Bridge/Debug dir not found."; exit 1; }
for dest in net40 net45; do
 cp *.dll *.xml ${brdhome}/lib/${dest}/.
 cp *.pdb ${brdhome}/lib/${dest}/pdb/.
done

# Builder
cd -
cd Builder/Build/bin/Debug || { echo "Builder/Debug dir not found."; exit 1; }
cp *.dll *.xml ${brdhome}/tools/.
cp Bridge.Build.dll ${brdhome}/tools/Bridge.Builder.dll

cd -

Although it's not a native windows script, it can help understanding how the NuGet package core files are distributed on an installed project.
Notice also the catch on the script about your observation:
"My guess is that the  tools\Bridge.Builder.dll  is out of sync with the other DLLs. However, it doesn't look like a new  Bridge.Builder.dll  was generated as part of the build."
I guess we should open a FAQ entry with "I've built Bridge but there's no Bridge.Builder.dll". :)
(I'm going to find a topic that I stressed this subject and share it here)

Here are the additional information about the "Bridge.Builder.dll vs Bridge.Build.dll" dillema:
Comment on jun 12th, issue #238

cp Bridge.Build.dll ${brdhome}/tools/Bridge.Builder.dll

Aha, this name might be what I missed. I searched for a dll of the same name. Will check this out next week. Thanks for the script!

Just installed VS2015 Community preview here and...
========== Build: 10 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========
No problems at all...

I didn't replace any DLLs in the repos I cloned, only in a test project I was trying to update.
I was using VS 2015 Enterprise RC. RTM came out today, so I will upgrade over the next few days and retry from a clean clone, and see how I get on.

@fabriciomurta, I see "10 succeeded" in your comment. I think @DanTup tested with Bridge.Frameworks.Builder.Testing.sln, but you did with Bridge.Frameworks.Builder.sln.
Those errors are reproducible with Bridge.Frameworks.Builder.Testing.sln only and it happens with the client and typeScriptBridge projects.
https://github.com/bridgedotnet/Bridge/issues/321#issuecomment-122295956

Yes, I didn't use the solution with the Testing project, following the guide to Build Bridge on http://bridge.net/kb/build-from-source/.

Yeah, sorry, I picked the testing one as I was hoping to run tests too. We'd like to ensure we can build the whole of Bridge and run tests in case we want to send PRs, or patch our own bug fixes in if required.

We have to get the Testing solution working on V 2015 as well and we are working on that. To be more specific, @LeonidVeriga is taking on that.

There are several differences in VS2015 compared to VS2013:

msbuild14 produces NoStdLib assemblies that cannot be read by Mono.Cecil. The following error comes up:
System.IndexOutOfRangeException: Index was outside the bounds of the array. at Mono.Cecil.Mixin.ParseRuntime(String self) at Mono.Cecil.PE.ImageReader.ReadMetadata() at Mono.Cecil.PE.ImageReader.ReadImage() at Mono.Cecil.PE.ImageReader.ReadImageFrom(Stream stream) at Mono.Cecil.ModuleDefinition.ReadModule(Stream stream, ReaderParameters parameters) at Mono.Cecil.ModuleDefinition.ReadModule(String fileName, ReaderParameters parameters) at ICSharpCode.ILSpy.LoadedAssembly.LoadAssembly(Object state) at System.Threading.Tasks.Task'1.InnerInvoke() at System.Threading.Tasks.Task.Execute()
The problem appears when reading RuntimeVersion field, it gets empty that is not handled by the Mono.Cecil code.

The following error appears when a "user's" project uses Bridge: Missing compiler required member 'System.String.Format' client C:\NET\Bridge\Testing\tests\client\CSC

Release configuration build fails with
ILMerge.Merge: ERROR!!: Duplicate type 'Mono.Security.Cryptography.CryptoConvert' found in assembly 'ICSharpCode.NRefactory.CSharp'. Do you want to use the /alllowDup option?	Bridge.Build

@DanTup can you please pull the changes and confirm the solution builds from source with no errors?

@LeonidVeriga Yep; all builds without issue on VS 2015 RTM now :+1:


Fix method visivility

This PR fixes method visibility after calling visibility changing methods like private, protected, public on receivers other than local context itself.
Motivation

Previously, methods defined after a private call would incorrectly be marked as private, even if the receiver is not self.
Also, it can be incorrect after calling methods that changes visibility likepublic, protected, and so on.
Like:
# typed: true

class Foo
  T.unsafe(nil).private

  def foo; end
end

Foo.new.foo

In this example, the foo method should not be marked as private, as the private call is not on self. However, method foo is marked as private and calling Foo.new.foo can not pass type checks.
> bundle exec srb tc
a.rb:9: Non-private call to private method foo on Foo https://srb.help/7031
     9 |Foo.new.foo
                ^^^
    a.rb:6: Defined in Foo here
     6 |  def foo; end
          ^^^^^^^
Errors: 1

In practice, code like a example below could not pass type checks.
class Model
  configure_something Settings.s3.private.bucket_name

  attr_accessor :my_variable
end

class Controller
  def action
    # ...
    model.my_variable = 1
    # ...
  end
end

Test plan

See included automated tests.

The VS Code job test failure is a transient network issue, not related to your change.


Download only certain chapters - royal road

Hi,
Thanks for the script. I'm enjoying reading my favourite web novels in epub format.
I'd like to know whether it is possible to download only a bunch of chapters instead of a whole book. Something like chapter 50 onwards. I need this because as the novels get updated daily and have reached chapter 400, I'd like to download just the chapters after 400. Is it possible? If so how do I pass the arguments?
Example site - https://www.royalroad.com/fiction/41618/mark-of-the-fool.
Thanks.

It's now possible as of 7230f65a689654d480c17426a9aa2fc34c752918. You just have to add the --offset option to the command. So e.g. leech https://www.royalroad.com/fiction/41618/mark-of-the-fool --offset 400 would start downloading from chapter 401.


question about database version

How to insert a new column to exist table?
We can update the database version within sqflite, but how to do within sqlcool?

this._db = await openDatabase(dbpath, version: 1,
onCreate: (Database _sqfliteDb, int version) async {
await _initQueries(schema, queries, _sqfliteDb, verbose);
}, onOpen: (Database _sqfliteDb) async {
// run create queries for file copied from an asset
if (fromAsset != null && checkCreateQueries) {
if (schema != null || queries.isNotEmpty) {
await _initQueries(schema, queries, _sqfliteDb, verbose);
}
}
});

The database migrations are not supported in Sqlcool


⚠ Fortnite has degraded performance

In 1f6ddae, Fortnite (https://fortnite.com) experienced degraded performance:

HTTP code: 200
Response time: 1670 ms

Resolved: Fortnite performance has improved in fdabb0f.


runtime: Change runtime version output format to fit dockerd

Description of problem
The error appears in dockerd,
dockerd[962]: time="2020-01-10T14:56:03.481652400+08:00" level=warning msg="failed to parse /usr/bin/kata-runtime version: unknown output format: kata-runtime : 1.10.0-rc0\n commit : 97d32c60e554306acb60d380242e4bedba600072\n OCI specs: 1.0.1-dev\n"
Expected result
According to moby parseRuntimeVersion function:
https://github.com/moby/moby/blob/d2d8e96f5163da77f31aad7ffb0e00ba99d682c9/daemon/info_unix.go#L234:6
kata-runtime version output should like this:
kata-runtime version 1.10.0-rc0
commit: 97d32c60e554306acb60d380242e4bedba600072
OCI specs: 1.0.1-dev

Actual result
kata-runtime  : 1.10.0-rc0
   commit   : 97d32c60e554306acb60d380242e4bedba600072
   OCI specs: 1.0.1-dev

I intend to work on the issue myself.


RAT-349: Fail the build if integration tests fail

As mentioned here https://github.com/apache/creadur-rat/pull/194#issuecomment-1890590107 if the integration tests fails the build does not fail.
The cause is that currently the build uses the integration-test goal which is documented as will never fail the build: https://maven.apache.org/plugins/maven-invoker-plugin/plugin-info.html
The run goal does the same thing and will let the build fail if one of the tests fails.

Thanks for the finding.


Fix memory leak in ShareActivity.java

@dbrant has explained that this is due to the activity context being passed into GpsExtractor. java on initiation. We should pass the application context instead if possible, or pass the context directly into the methods that need it.

Has this been fixed by the recent patch?

I didn't notice any leaks while uploading a couple of images. It would be great if others could help in testing this. @misaochan @neslihanturan need your help in testing it. :)

I tested (on a real device running 7.0) and don't get these leaks either, we'll see if @neslihanturan does.

Tested on a real device 4.2.4, and no leak occurred.

Closing the issue then. ShareActivity has undergone a lot of changes since the issue was reported. It might have got fixed in the process.


The number of apk package is too many

Hi, I faced with error from AppGallery a few day ago, do you have idea what about it and how to fix this?

Update App File Info is failed. Response: UpdateAppFileInfoResponse(ret=Ret(code=204144662, msg=[cds]add apk failed, additional msg is [90200503 The number of apk package is too many]))

@kuFEAR Hi! Have you tried to ask Huawei Support about this error?

@kuFEAR Hi! Have you tried to ask Huawei Support about this error?

no, i don't. ok I'll try and back with feedback

no, i don't. ok I'll try and back with feedback

@kuFEAR Hi! Do you have any news?

@kuFEAR Hi! Do you have any news?

Hi, when I’ve change deployType 'draft' to
upload-only' then error is gone, also I’ve remove a draft from huawei console, as I understand it has limitation on active drafts at the same time and will throw an error

Hi, when I’ve change deployType 'draft' to
upload-only' then error is gone, also I’ve remove a draft from huawei console, as I understand it has limitation on active drafts at the same time and will throw an error


EXC_BREAKPOINT raised when stopping stream on version 2.2.3

EXC_BREAKPOINT is raised when we stop the stream - causing the app to crash.
within iosrtcPlugin.swift - MediaStreamTrack_stop
	dispatch_async(self.queue) {
		pluginMediaStreamTrack!.stop()    <----- EXC_BREAKPOINT crash
	}

This does not happen in version 2.2.2 of the plugin.

Can you try changing the current code as follows?:

From:

func MediaStreamTrack_stop(command: CDVInvokedUrlCommand) {
	NSLog("iosrtcPlugin#MediaStreamTrack_stop()")

	let id = command.argumentAtIndex(0) as! String
	weak var pluginMediaStreamTrack = self.pluginMediaStreamTracks[id]

To:
func MediaStreamTrack_stop(command: CDVInvokedUrlCommand) {
	NSLog("iosrtcPlugin#MediaStreamTrack_stop()")

	let id = command.argumentAtIndex(0) as! String
	let pluginMediaStreamTrack = self.pluginMediaStreamTracks[id]

That worked. It no longer raises EXC_BREAKPOINT when closing the stream.

OK, will fix it.

Fixed in 21df2333ae2aab473d4a0d182a5f50215c5cada7

Thanks for this fix. Manually changing that code seems to do the trick, but the NPM repo still seems to have the version without these changes. Will that be updated anytime soon?

2.2.4 is out.


refactor(currencyconvert): use localstorage for rate

Describe the changes you have made in this PR

instead of using debounce and still do a rate-API call on every "view" (prompt, popup, etc) we cache the result in the browser-storage and only do an API-call if it's older than one minute
adjust all isLatestRate occurrences because those are not needed anymore

Link this PR to an issue
#1021
Type of change (Remove other not matching type)

refactor: New feature (non-breaking change which adds functionality)

Screenshots of the changes (If any)
none
How has this been tested?
Manually and via unit-tests
Checklist

[x] My code follows the style guidelines of this project and performed a self-review of my own code
[x] New and existing tests pass locally with my changes
[x] I checked if I need to make corresponding changes to the documentation (and made those changes if needed)

Does this also work when we want to show fiat rates in the notifications, too?
this is then probably another local storage then (which does not feel ideal to me)?


WG scripts no longer work

Scripts used to work, but no longer work
I keep getting
Server did not return OK. Stopping now

Here is the response that tech came back with.
Hello Benjamin,
Thank you for reaching out to PIA Support.
I understand that you're experiencing difficulties with the Wireguard script. To address this issue, I recommend reconfiguring your setup by using OpenVPN instead.

Issue on my setup. I think its working now.


What's the performance when rm_self_attn_decoder set to True?

Hello, I noticed that in https://github.com/IDEA-opensource/DAB-DETR/blob/94f20eda2be7cf38ddf94102db92e949b6846a65/models/DAB_DETR/transformer.py#L333 , there is a parameter rm_self_attn_decoder that is set to False by default.
Have you ever set it to True? What's the performance?

We found the performance droped a lot without self-attntention in decoder layers.


[Bug]: JSON Schema validation fails due to 'modified' format and null properties

What happened?
I tried running json schema validation tests on the OPDS 2.0 output, but they failed whenever publications were included.
Reason is that the 'modified' property should be of type 'string' format 'date-time' like 2018-11-13T20:20:39+00:00
But it's actually an object in the JSON output: {"date":"2012-03-03 19:47:50.000000","timezone_type":1,"timezone":"+00:00"}
See https://json-schema.org/understanding-json-schema/reference/string.html#dates-and-times for details.
OPDS validation error{
    "/publications/0/metadata/modified": [
        "The data (object) must match the type: string"
    ]
}

How to reproduce the bug
Try to validate the OPDS 2.0 output against the JSON schema using opis/json-schema or some other validator
Package Version
1.0.23
PHP Version
8.2
Which operating systems does with happen with?
Linux
Notes
No response

Sorry, a couple of other errors - it doesn't seem to like having null properties:
    ...
    "/publications/0/metadata/translator": [
        "The data (null) must match the type: string",
        "The data (null) must match the type: array",
        "The data (null) must match the type: object"
    ],
    "/publications/0/images/0/href": [
        "The data (null) must match the type: string"
    ],
    ...
    "/publications/1/metadata/belongsTo": [
        "The data (null) must match the type: object"
    ],
    ...

Yes, you can check out how I used the validator from opis/json-schema at
https://github.com/mikespub-org/seblucas-cops/blob/b20f22e4c86370a91aada1dab04ee75e49d1f8ed/test/KiwilanTest.php#L38
and which schemas I used for validation under https://github.com/mikespub-org/seblucas-cops/tree/main/test/schema
The relevant schemas were copied from the https://drafts.opds.io/ and https://readium.org/webpub-manifest/ sites until it stopped complaining :-)

And another "gotcha" that hasn't been pushed to GitHub yet: you need to specify setMaxErrors to see more than 1 error:
        self::$validator = new Validator();
        self::$validator->setMaxErrors(5);

Thanks for your help, I will release it today ;)

Yes, works for me - thanks :-)


Non-Civic Match has both SITE_NAME.partialMatch and OCCUPANT_NAME.partialMatch faults

Example:
https://delivery.apps.gov.bc.ca/pub/geocoder/occupants/addresses.xhtml?minScore=50&maxResults=10&addressString=PRINCE+RUPERT+LAW+COURTS--
Matches to "Prince Rupert Airport -- Digby Island, BC" with doubly-penalized partial name matches.

I suggest we close this as it is a duplicate of api-specs/201 and requires no work within the devkit to fix.

Duplicate of https://github.com/bcgov/api-specs/issues/201


Associate Issue with Project

This issue needs to be associated with Project ID 12 to ensure it is tracked and managed effectively. Please review the details and confirm the association.

Testing


P-values added to t-tests and wilcoxon test

I updated the rank_genes_groups function to output p-values for t-tests and the wilcoxon rank-sum test, as discussed with @falexwolf  in #159.
The changes are outlined below:

The t-test in the original file used a Welch t-test. I kept this, calculated the relevant degrees of freedom for a Welch test and then extracted the corresponding two-tailed p-value for the t-statistic (score).
The Wilcoxon test was originally done in chunks. To get the p-values I had to simplify this approach and use the ranksums function in scipy.stats. This caused me to loop through all of the genes being tested, which was fine for my dataset, but might need to be optimized for larger datasets.
The adjusted p-values (pvals_adj) were calculated with a standard Bonferroni correction.
All p-values are outputted and sorted the same way as 'names' or 'scores'. The only difference is that the p-values recarrays use float64 as a datatype to avoid converting a lot of the very small p-values to 0.

Hope this is helpful!
Andrés

Is it generally a good idea to output P-values for marker gene identification? As the null model is not valid for this setup, P-values are not really a measure of significance, but should only serve to order genes. I can see the point of outputting log fold changes, but outputting P-values may be misleading.

Why is the null hypothesis invalid? Testing whether two cell types have the same mean for the expression of a given gene seems valid, no? One-vs-rest might be problematic, though.

The null hypothesis assumes no difference in gene expression. However the groups are determined by differences in gene expression due to clustering. In random data, clustering would also separate clusters by 'random' gene expression signatures and thus find marker genes with p-value < 0.05.

Hmm, I think it's more complicated than that. First, cell type labels may not come from clustering. Even if it's the case, the design of Louvain does not rely on univariate mean difference between groups due to the distance metrics used in kNN graph, which take all genes into account at once.
Besides, I believe that how cell types are defined e.g. by biologist's manually annotation of each cell, Louvain, bulk comparison etc. doesn't really make the null hypothesis invalid. t-test just tests if the means are the same or not.
Regarding false positives with random clustering, that's why we have Bonferroni, no? But this is now a different story about p-values and multiple testing correction in general. So it's not our duty to solve it in rank_genes_group function. I'd rather prefer reporting a p-value (or maybe t-stat) and letting the user decide how valid/useful it is for his/her research instead of not outputting it with the consideration of the chance that it's not valid.

So you should easily be able to see that the null hypothesis is not valid by the distribution of p-values for all genes in one rank_genes_groups test. If it is valid, this distribution should be uniform. In that case it's only a multiple testing problem... I would guarantee you that it's not uniform though.
Cell type labels from expression-independent sources should not have the same confounding effect. However, I would bet you can trace back all biological annotations of cell types back to expression.

Thanks for your comments! This is an interesting discussion. I agree that the validity of a p-value depends on the user's data. From the standpoint of an analysis tool, I think it's better to report a p-value (and a t-statistic, which was already reported in the original function) and let the user decide what to do with that information.

I just noticed I forgot to answer to your point about multiple-testing-correction. Multiple testing correction deals with the effect that obtaining any false positives become more likely with repeated tests. However, it does not deal with bias. Bias is however present in the test given that the correct null hypothesis should assume some genes are different between groups, and not that all genes are equal. This comes back to the observation that some genes will be always be different between groups in random data, given that clustering defines the groups over which we test.
And as for louvain using whole transcriptome similarities rather than individual gene similarities... these are related. If you defined groups by differences of whole transcriptomes, this necesitates individual genes having different means. The correct p-value would be the output of a permutation test.
Could discuss this offline if you like @gokceneraslan. I'm preparing a manuscript which makes a point of this... if I am wrong, please convince me of this soon ;).

I can understand your line of reasoning @a-munoz-rojas. But is it not also dangerous to allow a user an interpretation which may be incorrect? I think outputting logFC values is great... I just have issues with calling the other output P-values. It is at the very least contentious whether they are measures of significance.

Sorry about the super late response and thank you for the PR, @a-munoz-rojas. I have not read most of the above discussion, yet. Right now, I just wanted to note that (@a-munoz-rojas), the tests are failing as the wilcoxon test seems to yield different results, now. I don't know whether it's just marginal, but it should be clarified; users should still get the same results as before.

Thanks for your reply, @falexwolf. I was looking through the same error and I can't really understand why the results are different - it might be a difference in accuracy levels between the manual wilcoxon method that was used before, and the built-in scipy.stats function I used. I looked at the differences and they indeed look marginal. I also compared a before-after with my dataset and I get very similar marker genes, albeit in slightly different order (see attached images). I don't really know what would be the best way to address these differences - I am simply using the built in spicy.stats function and not changing the output it gives me. Could this marginal difference be caused by the estimation in the "chunk" approach used in the previous version? Even with this marginal difference, I would assume that using the scipy function is more "accurate". Please let me know what you would prefer and what would be the best way to proceed.

Also, to reply to @LuckyMD, I definitely see your point. I was mostly trying to replicate other single-cell tools that do provide this metric, but I understand why it could be characterized as misleading - maybe we can add a disclaimer in the function description? I'm open to other suggestions!

@a-munoz-rojas Thanks for checking and thank you very much for the PR :smile! I like your version of the implementation a lot better than the manual one, as the code is much more readable. I don't know why @tcallies added the wilcoxon this way at the time, but I assume for speed and memory reasons.
So, I'm very happy to merge this PR; I'll just briefly give this another check today or tomorrow and update the tests so that they don't fail anymore.
Regarding the general discussion: Yes, let's just add a disclaimer that several assumptions on how meaningful the null hypotheses are both for wilxocon and t-test for single-cell data, should do the job. Then people will interpret the p-values with care.

So, the main problem I see is that the runtime of the new implementation is a factor 20 slower: 37 s vs. 2 s.

vs.

Let me think about what to do, here.

I'm merging this but will restore the previous Wilcoxon implementation, for speed reasons. The essential problem is that scipy.stats does not have a multi-dimensional implementation; it should be easy to adapt the previous implementation so that it provides pvalues, too; simply via multi-dimensional adaption of https://github.com/scipy/scipy/blob/v1.1.0/scipy/stats/stats.py#L4931-L4974.

If you @a-munoz-rojas would give it another try, using a multi-dimensional chunked implementation along the current implementation and the comment above, then that would be very cool! If you don't have bandwidth for that, could you pass this on to @Koncopd, who might have bandwidth? It would be nice to clean up the whole module (in particular, split up the long code chunks into three functions _t_test(), _wilcoxon(), _logreg()).
I also made the tests work again; I guess they failed as the current implementation and scipy are handling ties a little different; the results for scores were exactly the same up to the position of a single gene.
Please make a new PR for any of this.

Ah, now we just wrote at the same time. ;) Let me know what you think.

Ok, now the tests are actually passing again, everything is in the three commits prior and including this one: https://github.com/theislab/scanpy/commit/d889faf9a58d8981c0783584b3f333680fc161ce

And: just as a remark; the chunking along genes is necessary so that everything still fits into memory for really large datasets, which are almost always sparse when fed into rank_genes_groups.

I was just about to ask about the chunking along genes - you read my mind @falexwolf. I think it might be possible to do a multi-dimensional adaptation of the scipy.stats code you linked to, and still do the math with sparse matrices, similar to how we implemented the t-tests. This way we could possibly avoid the chunking (it might help with readability of the code). Would this be worth pursuing?
I'll give this a quick try, but I am a little limited in bandwidth. I'll let you know soon if it would be best to get some help from @Koncopd (if they have time!)


Illegal constructor when extending DocumentFragment

Extending DocumentFragment is used by @github/template-parts and document-persistent-fragment
Using these in linkedom causes Illegal constructor error being thrown, even though the browser and JSDOM handles this just fine.

JSDOM pollutes the global or assume there's only one document on the global, which is nonsense in here ... almost every class in here requires the ownerDocument as first argument so you can bind the class passing your document reference and be good with it.
P.S. document-persistent-fragment is not needed anymore, I've found a way better solution in latest uhtml library and I might soon just publish that as module a part where no global patch is ever needed.


DenseNet generated image embeddings have very small variance and are sparse

I'm working with pretrained DenseNet161 model to learn and generate image embeddings.
I consider output of adaptive_avg_pool2d layer image embedding. I noticed that it's very sparse and has small variance. If we consider a batch of 64 images, variance of the same feature over all images in a batch is ~2e-6. Also the feature values are just very small (~1e-3). This is for pretrained model, if I start fine-tuning the model, some features quickly become 0 across all images in the batch.
Surprisingly, this is not the case for resnet. Running the same training for resnet50 results in expected features of size ~[0, 2] and variance around 1.
This is stunning difference for me. Not sure if there's some problem with implementation or this is just DenseNet works. Please let me know if  you have similar experiences.

Hi,
I have never used DenseNet myself, so I wouldn't know the answer to your question.
I would first check though that your densenet model is outputting correct class predictions -- maybe you are passing the wrong input normalization, which makes almost all the activations go to zero due to ReLU?

我正在使用预训练的DenseNet161模型来学习和生成图像嵌入。
我考虑了adaptive_avg_pool2d图层图像嵌入的输出。我注意到它非常稀疏且变化很小。如果我们考虑一批64张图像，则同一特性在一批图像中的方差约为2e-6。同样，特征值也很小（〜1e-3）。这是针对预训练模型的，如果我开始对模型进行微调，则批处理中所有图像上的某些功能会很快变为0。
令人惊讶的是，resnet并非如此。对resnet50进行相同的训练会导致预期特征的大小约为[0，2]，方差约为1。
这对我来说是惊人的区别。不确定实现是否存在问题，或者这仅仅是DenseNet的工作。如果您有类似的经验，请告诉我。

Hello, I'm learning densenet121on Imagenet, the test accuracy after importing the pre training parameters is about 0. Are you referring to the pre training weight parameters in the article?

@filoliufei are you using the image normalization as expected by the model? Also, do you get expected accuracies with ResNet50?

Yes. The problem has been solved. Thank you.

This issue appears to be stale, so I'm closing it. If you think it should remain open, feel free to reopen it.

This issue appears to be stale, so I'm closing it. If you think it should remain open, feel free to reopen it.


refactor(react-router): Import only necessary components

Instead of importing the entire library, cherry-pick the components we use.
This is supported and encouraged by react-router.
Reduces the size of the main bundle by 9 kB (709 kB -> 700 kB) - we are no longer implicitly importing history/createHashHistory and history/createMemoryHistory.

Coverage remained the same at 99.242% when pulling 61b919e54221e1e75c4ca9565583102e57f8415f on sorin-davidoi:import-minimal-react-router into 5bd8d7ab8cc4de0afe4b9c259e2abeaf7503cbf2 on react-boilerplate:dev.

I thought tree-shaking feature of webpack 2 kicks in and the unwanted modules are stripped down.. 🤔

@KarandikarMihir From my limited understanding:
import { applyRouterMiddleware, Router, browserHistory } from 'react-router';
Is effectively importing all of react-router and then, using ES6 destructuring to assign applyRouterMiddleware, Router and browserHistory to what are effectively local constants.
(I know that does not agree with orthodox thought, but the the concept of 'Hoisting' is another example of orthodox thought making an unnecessary abstraction over the simplicity of a two-pass compile process where vars are processed first, then statements).
In any case, I believe tree shaking can only eliminate library code that is not (explicitly or implicitly) imported.
@sorin-davidoi Could you post some before and after charts using, for example webpack-visualizer ?
See: #1309

@KarandikarMihir From what I understand, tree-shaking only applies to ES6 modules, while most libraries ship as CommonJS modules (see "The Underlying problem" section from here).
Attaching screenshots obtained with webpack-bundle-analyzer and webpack-visualizer.
Before:

After:

LGTM :sparkles:

@gihrig

'Hoisting' is another example of orthodox thought making an unnecessary abstraction over the simplicity of a two-pass compile process.

I couldn't agree more.
@sorin-davidoi Well, this was a good learning experience for me. Awesome! :sparkles:

Coverage remained the same at 99.242% when pulling b516037d4b352bebc38508259697773f170b5dec on sorin-davidoi:import-minimal-react-router into 4613da7a3e64d6ab093e16749742c7bdca52af04 on react-boilerplate:dev.

Coverage remained the same at 99.242% when pulling b516037d4b352bebc38508259697773f170b5dec on sorin-davidoi:import-minimal-react-router into 4613da7a3e64d6ab093e16749742c7bdca52af04 on react-boilerplate:dev.

AppVeyor and Chrome are not seeing eye-to-eye...

Download of googlechromestandaloneenterprise64.msi (48.26 MB) completed.
Error - hashes do not match. Actual value was '1450DE87F0289CE725DF1C138BBA2666304FB94BF4476B03C06D058C999D4805'.
ERROR: Checksum for 'C:\Users\appveyor\AppData\Local\Temp\1\chocolatey\GoogleChrome\56.0.2924.87\googlechromestandaloneenterprise64.msi' did not meet 'c47ff551ff8b251268905cd87fb299959e6bcb3640a3e60085a3a7dbf176845f' for checksum type 'sha256'. Consider passing --ignore-checksums if necessary.
The install of googlechrome was NOT successful.
Error while running 'C:\ProgramData\chocolatey\lib\GoogleChrome\tools\chocolateyInstall.ps1'

I think there was a similar problem with #1611, which @tmf found a way around.

Coverage remained the same at 99.242% when pulling b9f21c2247d3d7970b0b23b9e6cd1952e0e86a5f on sorin-davidoi:import-minimal-react-router into c855f8981e86e7cbbd8c526013646567bba253a1 on react-boilerplate:dev.

Coverage remained the same at 99.242% when pulling 22c8c182620e3bb240647a8c5252df931a219050 on sorin-davidoi:import-minimal-react-router into 04c0687414b739457d8067ec0fbcf9502f25b17c on react-boilerplate:dev.

Coverage remained the same at 99.242% when pulling 998d4e79431df8143d3d49a6c4dcfd8711ca3a6a on sorin-davidoi:import-minimal-react-router into 5c495e7185ab7f584cb4008acd8258a114b41648 on react-boilerplate:dev.

Coverage remained the same at 99.242% when pulling 4d669de33e5a5f5c4db92f237daf32ad1317c937 on sorin-davidoi:import-minimal-react-router into ea1e1c451c12bd4b36057d884a1087d45a27310a on react-boilerplate:dev.

Awesome, merging this! @sorin-davidoi Thanks for all your work :)