kconfig for lvgl_esp32_drivers/lvgl_i2 causes ESP-IDF SDK Configuration Editor to Crash

I've updated to the latest master of lvgl_esp32_drivers.
I'm using LVGL8.3 master.

I can select i2C port Zero, but in the KConfig specific to I2C Port Settings,
if I select Enable I2C port 0 or Enable I2C port 1, instead of bringing up the options to set the pins etc, it just crashes menuconfig.
Please, I don't know how to set my pins.
Also, when including this line of code as per the readme before calling lv_init():
lvgl_i2c_locking(i2c_manager_locking());

I get the following compiler error:

implicit declaration of function 'i2c_manager_locking'; did you mean 'i2c_master_write'? [-Werror=implicit-function-declaration]

In the sdkconfig, the following [ CONFIG_LV_I2C=y and CONFIG_LV_I2C_DISPLAY_PORT=0] appears to be set, but it is not selected anywhere in menuconfig:
'''
CONFIG_LV_DISP_BACKLIGHT_OFF is not set
CONFIG_LV_DISP_BACKLIGHT_SWITCH=y
CONFIG_LV_DISP_BACKLIGHT_PWM is not set
CONFIG_LV_BACKLIGHT_ACTIVE_LVL is not set
CONFIG_LV_DISP_PIN_BCKL=-1
CONFIG_LV_I2C=y
CONFIG_LV_I2C_DISPLAY_PORT=0
end of LVGL TFT Display controller
'''
and in the kconfig file for lvgl_esp32_drivers/lvgl_tft/kconfig:
line 996
'''
config LV_I2C
bool
default y if LV_I2C_DISPLAY
config LV_I2C_DISPLAY_PORT
    int
    default 1 if LV_I2C_DISPLAY_PORT_1
    default 0

'''
And  at line
177
'''
Display controller communication protocol
#
# This symbols define the communication protocol used by the
# ESP32 to communicate with the display controller.
# This symbols can be used, but not limited, to:
# - Know what peripherals to initialize.
# - Know if the touch and display controllers share the same peripheral.
# - Etc.
config LV_TFT_DISPLAY_PROTOCOL_SPI
    bool
    help
        Display controller protocol SPI

config LV_I2C_DISPLAY
    bool
    help
        Display controller protocol I2C

'''
But I'm using the ILI9488, and thus
'''
config LV_TFT_DISPLAY_USER_CONTROLLER_ILI9488
bool "ILI9488"
select LV_TFT_DISPLAY_CONTROLLER_ILI9488
select LV_TFT_DISPLAY_PROTOCOL_SPI
'''
but this does not exclusively set SPI, so LV_I2C=y is also set????

I resorted to screen recording the error that flashes on the screen momentarily in VSCode using the ESP-IDF plugin, when the menuconfig crashes.
The error was a result of the int value not having a default, and thus ''
Fixing this allowed for the proper menuconfig handling of the I2C ports
if I2C_MANAGER_0_ENABLED
		config I2C_MANAGER_0_SDA
			int "SDA (GPIO pin)"
			default 0
		config I2C_MANAGER_0_SCL
			int "SCL (GPIO pin)"
			default 0

and this change is required for both ports 0 and 1.
Now all I need to do is get past the
implicit declaration of function 'i2c_manager_locking'; did you mean 'i2c_master_write'? [-Werror=implicit-function-declaration] error

@ropg could you please have a look?

Will do as soon as I find some time.

Ok, let's close this, feel free to open it again if you have issues.


NIFI-3490 added SAN option for TLS toolkit in standalone mode

Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[X] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[X] Has your PR been rebased against the latest commit within the target branch (typically master)?

[X] Is your initial contribution a single, squashed commit?

For code changes:

[X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[X] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

Test example:
./tls-toolkit.sh standalone -s "nifi.vip.org,localhost,nifi-01" -n nifi-01

$ keytool -list -keystore keystore.jks -v

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: nifi-key
Creation date: Feb 22, 2017
Entry type: PrivateKeyEntry
Certificate chain length: 2
Certificate[1]:
Owner: CN=nifi-01, OU=NIFI
Issuer: CN=localhost, OU=NIFI
Valid from: Wed Feb 22 22:25:30 CET 2017 until: Sat Feb 22 22:25:30 CET 2020
Certificate fingerprints:
...

Extensions:

#1: ObjectId: <IP_ADDRESS> Criticality=false
...

#2: ObjectId: <IP_ADDRESS> Criticality=false
...

#3: ObjectId: <IP_ADDRESS> Criticality=false
...

#4: ObjectId: <IP_ADDRESS> Criticality=true
...

#5: ObjectId: <IP_ADDRESS> Criticality=false
SubjectAlternativeName [
  DNSName: nifi.vip.org
  DNSName: localhost
  DNSName: nifi-01
]

#6: ObjectId: <IP_ADDRESS> Criticality=false
...

@pvillard31 I have limited connectivity for the next couple days but I am happy to review this as I reviewed your similar changes for client/server mode.

Thanks @alopresto, really appreciated. In particular I'd like your opinion regarding the option letter: I used N because both s and S are already used in standalone mode. Should I change on client side to have the same letter on both sides?

In general I would push to keep the flags consistent across client/server and standalone mode. On the other hand, I am hesitant to change a flag that already exists in the wild (although, it was just committed to master; it's not in a release yet).
I guess talking through that to myself, I would vote for "change the client/server flag to be consistent with standalone" but I would also welcome input from @brosander and @joewitt .

I agree that changing flags is likely to cause confusion although since we haven't had a release yet, I would hope it would be minimal impact. Consistency across modes of operation will help reduce surprise which is always a good thing.
Do we expect this to be something most users specify or is it a more advanced usecase?  If it's more advanced and less common then a short version of the argument may not be necessary.

That's a good point; not every flag needs a short code. SANs, while useful, are definitely less common than some of the other features of the toolkit.

I verified the contrib-check, all tests pass, smoke tests on flows work, and the tool now generates SANs in standalone mode.
#5: ObjectId: <IP_ADDRESS> Criticality=false
SubjectAlternativeName [
  DNSName: nifi.vip.org
  DNSName: localhost
  DNSName: nifi-01
]

If you remove the short code or change the client/server short code to match, I'll +1 and merge.

Hey guys, hopefully I've addressed all your comments, thanks for the review!

changed the option to SAN
refactored code as suggested by @olegz

Thanks @pvillard31 . For the record, I know some people prefer the Java 8 lambdas, and I come from a Ruby/Groovy background where I use closures and chained collection operations for everything, but there can be pitfalls to replacing concise traditional Java for:each loops with lambdas (see performance issues and other concerns) in addition to providing a learning curve for future developers who are not familiar with the construct.
I'm not going to request changing this back, and after I do a final review & smoke test, I'll merge this, but in the future I don't think we need to be as aggressive about ensuring new code always uses the newest feature when the existing process is fine.

@pvillard31 I think I was a little unclear -- I didn't expect both -SAN and --subjectAlternativeNames, but just the long flag --subjectAlternativeNames. Some commands stack their short flags (i.e. -S and -A run on the same command could also be expressed -SA so all short flags must only be one character).
I ran the standalone tool with both -S and -SAN on the same line and it worked but it strikes me as potentially confusing for users.

Sorry for the misunderstanding @alopresto, agreed that it could be confusing. Updated the PR as per your comments. Should be good now.

@pvillard31 I reviewed this again. I like it; my only concern is that if you run with -SAN it will still process, it won't show any error and a user might expect it to generate a SAN in the keystore though it won't. As no one other than the commenters on this PR have experienced this, however, I do not think it should stop the inclusion of this feature.
hw12203:...assembly/target/nifi-toolkit-1.2.0-SNAPSHOT-bin/nifi-toolkit-1.2.0-SNAPSHOT (pr1530) alopresto
🔓 1224747s @ 16:32:49 $ ./bin/tls-toolkit.sh standalone -n localhost -O -S password -SAN hostname.com
2017/03/06 16:33:04 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandaloneCommandLine: No nifiPropertiesFile specified, using embedded one.
2017/03/06 16:33:04 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Running standalone certificate generation with output directory ../nifi-toolkit-1.2.0-SNAPSHOT
2017/03/06 16:33:04 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Using existing CA certificate ../nifi-toolkit-1.2.0-SNAPSHOT/nifi-cert.pem and key ../nifi-toolkit-1.2.0-SNAPSHOT/nifi-key.key
2017/03/06 16:33:04 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Overwriting any existing ssl configuration in ../nifi-toolkit-1.2.0-SNAPSHOT/localhost
2017/03/06 16:33:04 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Successfully generated TLS configuration for localhost 1 in ../nifi-toolkit-1.2.0-SNAPSHOT/localhost
2017/03/06 16:33:04 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: No clientCertDn specified, not generating any client certificates.
2017/03/06 16:33:04 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: tls-toolkit standalone completed successfully
hw12203:...assembly/target/nifi-toolkit-1.2.0-SNAPSHOT-bin/nifi-toolkit-1.2.0-SNAPSHOT (pr1530) alopresto
🔓 1224763s @ 16:33:05 $ keytool -list -v -keystore localhost/keystore.jks -storepass password

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: nifi-key
Creation date: Mar 6, 2017
Entry type: PrivateKeyEntry
Certificate chain length: 2
Certificate[1]:
Owner: CN=localhost, OU=NIFI
Issuer: CN=localhost, OU=NIFI
Serial number: 15aa62f097a00000000
Valid from: Mon Mar 06 16:33:04 PST 2017 until: Thu Mar 05 16:33:04 PST 2020
Certificate fingerprints:
	 MD5:  52:CF:22:54:02:AB:22:8E:DE:AC:C8:2E:3F:8C:1B:2C
	 SHA1: 55:87:B5:20:8F:1F:03:F3:D2:68:85:F5:4E:49:85:D5:53:6A:27:11
	 SHA256: E1:6E:F6:89:73:70:26:31:57:CB:8B:E6:44:DA:32:0B:77:39:22:1D:EA:5E:B8:3E:2D:4F:24:4C:68:2A:A4:3F
	 Signature algorithm name: SHA256withRSA
	 Version: 3

Extensions:

#1: ObjectId: <IP_ADDRESS> Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: D0 B6 50 54 15 F2 64 EA   AA EB D4 82 A0 07 B4 2D  ..PT..d........-
0010: 28 AC 66 CF                                        (.f.
]
]

#2: ObjectId: <IP_ADDRESS> Criticality=false
BasicConstraints:[
  CA:false
  PathLen: undefined
]

#3: ObjectId: <IP_ADDRESS> Criticality=false
ExtendedKeyUsages [
  clientAuth
  serverAuth
]

#4: ObjectId: <IP_ADDRESS> Criticality=true
KeyUsage [
  DigitalSignature
  Non_repudiation
  Key_Encipherment
  Data_Encipherment
  Key_Agreement
]

#5: ObjectId: <IP_ADDRESS> Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: D0 DC A2 03 C9 18 03 B8   B1 1B 0E 11 BC A5 A7 CF  ................
0010: 1F 68 3A 63                                        .h:c
]
]

Certificate[2]:
Owner: CN=localhost, OU=NIFI
Issuer: CN=localhost, OU=NIFI
Serial number: 15aa62eb59a00000000
Valid from: Mon Mar 06 16:32:43 PST 2017 until: Thu Mar 05 16:32:43 PST 2020
Certificate fingerprints:
	 MD5:  44:15:FC:42:BE:A3:A5:7E:C3:86:AF:82:50:51:E3:E4
	 SHA1: 38:D3:42:19:6C:71:5C:02:BF:8E:A1:02:DE:A3:D8:0C:D4:73:8D:B7
	 SHA256: 13:71:F8:1A:22:A3:43:93:29:1D:2F:41:F8:C0:1E:25:79:E2:D7:5D:28:53:5C:21:97:A0:68:6C:AD:39:18:62
	 Signature algorithm name: SHA256withRSA
	 Version: 3

Extensions:

#1: ObjectId: <IP_ADDRESS> Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: D0 B6 50 54 15 F2 64 EA   AA EB D4 82 A0 07 B4 2D  ..PT..d........-
0010: 28 AC 66 CF                                        (.f.
]
]

#2: ObjectId: <IP_ADDRESS> Criticality=false
BasicConstraints:[
  CA:true
  PathLen:2147483647
]

#3: ObjectId: <IP_ADDRESS> Criticality=false
ExtendedKeyUsages [
  clientAuth
  serverAuth
]

#4: ObjectId: <IP_ADDRESS> Criticality=true
KeyUsage [
  DigitalSignature
  Non_repudiation
  Key_Encipherment
  Data_Encipherment
  Key_Agreement
  Key_CertSign
  Crl_Sign
]

#5: ObjectId: <IP_ADDRESS> Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: D0 B6 50 54 15 F2 64 EA   AA EB D4 82 A0 07 B4 2D  ..PT..d........-
0010: 28 AC 66 CF                                        (.f.
]
]

*******************************************
*******************************************

hw12203:...assembly/target/nifi-toolkit-1.2.0-SNAPSHOT-bin/nifi-toolkit-1.2.0-SNAPSHOT (pr1530) alopresto
🔓 1224775s @ 16:33:17 $

Reviewed, verified contrib-check and all tests, and ran tool. Merged.


.cargo/config contains paths that need to be adjusted for tusk build:prod

.cargo/config contains several references to  /Users/rousan, that will not work on any other users machine.
This should be mentioned in the README

To be able to build for arm64 it is also necessary to increase the ndk to (at least?) 23.1.7779620


Circular progress bar with color change based on value

is it possible to change the color based on value with an animation like this?
https://codepen.io/matousek/pen/LZEyQa

Wow nice feature i like it !

finally landed the feature https://github.com/nithinpp69/react-native-circular-progress-indicator/tree/develop#animated-stroke-color. Please check the API and let me know your thoughts.

finally landed the feature Animated Stroke Color Please check the API and let me know your thoughts.

Amazing thanks a lot ♥.

New version landed on NPM V4.3.0.


Update Import CSV

WIP - I'm working on making the import function a bit more versatile. Now to add a new field, you just need to add it to the ValidFields interface and add a callback (hopefully already defined in the caseFunctions.ts file) to the callbacks object. Let me know if you guys think this is better than it was. I set it up so it can now import math-schedule.csv mostly successfully. I just still haven't set it up to parse duration or semesterLength.

Upon thinking about this, I have a couple thoughts:

I think overall this change is worth it, due to the gains in maintainability.
The one area where I am slightly concerned is that this code no longer has a concept of preference between similar headers, so for example if the headers startTime and startTimeStr both exist in a CSV, this code will just utilize the data in whichever field appears last (and I believe it parses both fields) instead of preferring startTimeStr. Now since we are likely going to opt for a standardized format for the schedules, this may not matter, but worth keeping in mind this slight loss of functionality.


Disable  drag Close left or right

how to disable the close on drag left or right?

If I'm not wrong, this feature is not implemented. As you can see here, there is a flag you can enable or disable to configure the alpha effect applied during the horizontal drag. But the feature you request is not implemented.


No help for "add corner profile for"

Describe the bug
No help for "add corner profile for"
Steps to reproduce the behavior
1.Navigate to https://arcade.makecode.com/stable--skillmap#sc
2.Right-click on "add corner profile for"
Actual behavior
No help for "add corner profile for"

Additional context
1.OS: Windows(rs6)
2.arcade version: 1.6.4
3.Microsoft MakeCode version: 7.2.6

Same issue for "make mySprite gravity jump"

These are blocks created right in the document, so the best I can do is add rollover text, which I've done.  You should be able to see that live now.

@kiki-lee  Is this what rollover text is referring to?


matern12 is wrong!

There is a sqrt missing from the distance.
It should be
def matern12(x,xpr,l):
    """Matern 1/2 base function"""
    fac2 = np.sqrt((x-xpr)*(x-xpr))
    return np.exp(-fac2/l)

Not
def matern12(x,xpr,l):
    """Matern 1/2 base function"""
    fac1 = (x-xpr)*(x-xpr)
    return np.exp(-fac1/l)

I just got bitten by this ... again, @iosonobert could you please do something about it, merge #19 for example?
cheers


libsForQt5.qwt: 6.1.6 -> 6.2.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/qwt/versions.
meta.description for libsForQt5.qwt is: "Qt widgets for technical applications"
meta.homepage for libsForQt5.qwt is: "http://qwt.sourceforge.net/"
meta.changelog for libsForQt5.qwt is: ""
Updates performed

Version update

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS

The tests defined in passthru.tests, if any, passed

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 6.2.0 with grep in /nix/store/9mznp6nf879mzr1s5i3748gpchgcwcpk-qwt-6.2.0

found 6.2.0 in filename of file in /nix/store/9mznp6nf879mzr1s5i3748gpchgcwcpk-qwt-6.2.0

directory tree listing: https://gist.github.com/dc38b9841b2c00ee68044f6b672ed0d7

du listing: https://gist.github.com/3a5a20c6c59a1258988259b214fca263

Rebuild report (if merged into master) (click to expand)

90 total rebuild path(s)

25 package rebuild(s)

25 x86_64-linux rebuild(s)
25 i686-linux rebuild(s)
15 x86_64-darwin rebuild(s)
25 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
caneda
gnss-sdr
gnuradio
gnuradio3_8
gnuradio3_8Packages.ais
gnuradio3_8Packages.gnuradio
gnuradio3_8Packages.limesdr
gnuradio3_8Packages.osmosdr
gnuradio3_8Packages.rds
gnuradioPackages.gnuradio
libsForQt5.qwt
libsForQt512.qwt
libsForQt514.qwt
libsForQt515.qwt
linssid
opl3bankeditor
opn2bankeditor
plasma5Packages.qwt
pothos
qgis
qgis-unwrapped
qradiolink
qstopmotion
sonic-pi
testdisk-qt

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/9mznp6nf879mzr1s5i3748gpchgcwcpk-qwt-6.2.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A libsForQt5.qwt https://github.com/r-ryantm/nixpkgs/archive/d3ed6d63fbb9eb3fbc45f2c82f93d9670a7d3f7e.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/9mznp6nf879mzr1s5i3748gpchgcwcpk-qwt-6.2.0
ls -la /nix/store/9mznp6nf879mzr1s5i3748gpchgcwcpk-qwt-6.2.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

17 packages failed to build:

caneda
gnss-sdr
gnuradio
gnuradio3_8
gnuradio3_8Packages.ais
gnuradio3_8Packages.gnuradio
gnuradio3_8Packages.limesdr
gnuradio3_8Packages.osmosdr
gnuradio3_8Packages.rds
gnuradioPackages.gnuradio
linssid
opl3bankeditor
opn2bankeditor
pothos
qgis
qgis-unwrapped
qradiolink

6 packages built:

libsForQt5.qwt
libsForQt512.qwt
libsForQt514.qwt
qstopmotion
sonic-pi
testdisk-qt

Maintainer pings
cc @bjornfor for testing.

Result of nixpkgs-review pr 137804 at d3ed6d63 run on aarch64-linux 1

1 package marked as broken and skipped:

sonic-pi

6 packages failed to build:

caneda
gnuradio
gnuradioPackages.gnuradio
linssid
opl3bankeditor
opn2bankeditor

11 packages skipped due to time constraints:

gnss-sdr
gnuradio3_8
gnuradio3_8Packages.ais
gnuradio3_8Packages.gnuradio
gnuradio3_8Packages.limesdr
gnuradio3_8Packages.osmosdr
gnuradio3_8Packages.rds
pothos
qgis
qgis-unwrapped
...

5 packages built successfully:

libsForQt5.qwt (libsForQt515.qwt ,plasma5Packages.qwt)
libsForQt512.qwt
libsForQt514.qwt
qstopmotion
testdisk-qt

1 suggestion:

error: no-flags-spaces
qmakeFlags cannot contain spaces, please use qmakeFlagsArray in Bash.
Near pkgs/development/libraries/qwt/6.nix:18:3:
   |
18 |   qmakeFlags = [ "-after doc.path=$out/share/doc/${name}" ];
   |   ^

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement. Does something look off? Please file an issue or reach out on IRC.

Closed by #165335.


support for eslint no-restricted-syntax

Hi,
we actively using custom selectors with no-restricted-syntax eslint rule.
Are the any plans to support this feature in the nearest future?
Thanks!

We are unable to support this rule. Try https://github.com/ast-grep/ast-grep instead.
Which syntax are you trying to restrict? We have a few implementations such as https://oxc.rs/docs/guide/usage/linter/rules/oxc/no-rest-spread-properties.html or https://oxc.rs/docs/guide/usage/linter/rules/oxc/no-optional-chaining.html

Hi @Boshen!
Thanks you for the fast reply 🙇
currently we using underscore detection rule for the import, i.e.
"selector": "ImportSpecifier[imported.name=/^_/]"
ast-grep seems a really good fit for that task, thanks.


Add ability to pull from CommunityDragon

It would be neat if you could configure DragonService to be able to pull from the cdragon project instead of ddragon, since ddragon has known data integrity issues.

Probably will wait until there is a stable CDN for the assets and hopefully a way to deploy it yourself without having to rely on their webhost.


Install on VSC

Installing on the VSC remains a nightmare.
On the VSC virtual envs and conda are discouraged.
There is VSC documentation  on installing Python packages alongside software modules.
But! This requires that the already installed packages which are also required by pyfa meet the requirements. This is often not the case. e.g. the standard R module (required for Rfa) comes with:
scipy 1.8.1 requires numpy<1.25.0,>=1.17.3, but you have numpy 1.26.2 which is incompatible.
poetry 1.1.13 requires packaging<21.0,>=20.4, but you have packaging 23.2 which is incompatible.

which is incompatible.
Therefore, when using as recommended in the documentation of the VSC
 pip install  --prefix="${VSC_DATA}/python_lib/pyfa/" --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pyfa --upgrade

pip attempts to uninstall these and fails:
 Attempting uninstall: packaging
    Found existing installation: packaging 20.9
    Uninstalling packaging-20.9:
ERROR: Could not install packages due to an OSError: [Errno 30] Read-only file system: 'LICENSE.APACHE'

Current workaround:
This seems to be avoided when not using the --prefix option
pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pyfa --upgrade

Thus installing the package despite providing a dependency error:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
scipy 1.8.1 requires numpy<1.25.0,>=1.17.3, but you have numpy 1.26.2 which is incompatible.
poetry 1.1.13 requires packaging<21.0,>=20.4, but you have packaging 23.2 which is incompatible.

Pyfa is usable but installed on scratch directory in .local. Not ideal.
Possible solutions

Use the workaround and move pyfa to a more permanent location
Find a module that exactly satisfies the pyfa needs or change the requirement needs of pyfa
Install pyfa (for a given module set) in a shared data location and simply add the path to your python path. Similar to how daan installed meteogrid on VSC.

@kobebryant432  Thanks for exploring this!
Firstly i will try to cleanup the packaging (Fix dependency errors, and move the package to PyPi). Possibly i have to rename this project to PyFa-tool (feel free to give your better suggestion), since pyfa is already taken. If that is done (i am working on it now), than you can test the installation again.
If the issue remains, than i suggest to try a install from source. So to clone this repo to vsc, or to copy the .tar file to vsc, and install it with pip.
I will keep you posted on the status of the deployment to PyPi

@kobebryant432 the current version (v0.0.1) is deployed to PyPi (see the readme). Could you this a try on VSC:
pip install PyFa-tool --upgrade
installation?

I tried this on the VSC but I still have the same issues. I think this is a VSC issue. It may be a good idea to ask them if they could install (and maintain) a version of PyFa in a module or support us with the install. An easy install is essential as most of the potential users will be working on the VSC.

Or, one of us could host a copy of this repo on the VSC machines and add the path to the user's pythonpath. In that way, we have some more control over switching versions etc, or fixing bugs.
We could ask the support of VSC to set this up.
@kobebryant432 what do you think?

This is something I am still struggeling with. I tried to redo the install (as a test for other users) but it is not clean. Current install is on .local (which is always on path). In addition, R is required (so a VSC module is needed) but this brings along python packages that conflict with the required packages.
@vergauwenthomas Can you update PyPi with the newest version?

@kobebryant432, is it possible to use pip and istall it from github? I found this more convenient if changes to the package are required.
BTW. I am currently working on the installation of PyFa on the ECMWF Atos machine, i will send you my notes how a did it. Maybe that can help you.

If you can share your notes that would already be great.

Here are my notes:
Installation on ECMWF atos is done by using as mutch modules as possible.
#1 Setup python environment
Create a .sh file with:
module purge
module load pi #for an intel env
module load python3/3.10.10-01
module load gdal/3.6.2
module load git/2.39.1
module load rstudio
#2 Use python venv
setup a venv:
python3 -m venv /home/cu1c/software/py_envs/pyfa_env
activate the environment:
source /home/cu1c/software/py_envs/pyfa_env/bin/activate
#3. Install dependencies
pip install pandas
pip install matplotlib
pip install xarray
pip install cartopy
pip install netcdf4
pip install rioxarray
pip install geopandas
#4. install the pyfa CLI
go to the repo folder
cd /home/cu1c/software/PyFa
Open python and run
python3
import pyfa_tool as pyfa
pyfa.setup_shell_command()
#5 test installation
pyfa -h
..... output ....
go to a FA file and try to describe, plot and convert

If you can download the repo to VSC, then create a new branch which is a copy of the latest #54  branch. You can add fixes in there if they are needed for the installation.

I will try this!

@kobebryant432
Als je wat tijd over zou hebben zou je misschien beter het laden van de juiste modules zelf beheren. Nu wordt dit weggeschreven in de bashrc van iedere gebruiker. Stel dat we in de toekomst nieuwe versies van de modules nodig hebben, dan moet iedere gebruiker zijn/haar bashrc updaten.
Mijn suggestie is om een .sh file te maken pyfa_on_vsc_setup.sh met deze inhoud:
#!/usr/bin/env bash

module load Python/3.10.4-GCCcore-11.3.0
module load Tkinter/3.10.4-GCCcore-11.3.0
module load GDAL/3.5.0-foss-2022a
module load R/4.2.1-foss-2022a

#Load the v-env
 source /dodrio/scratch/projects/starting_2022_075/Software/PyFa/pyfa_env/bin/activate

Zo heb jij wat meer touwtjes in handen van de vsc gebruikers als er problemen zijn met modules/packages etc

@vergauwenthomas Yes agreed, this is a better solution. I will do this on Monday, test it and update the readme! :)


High-level category counts do not respond to filters

Example:

The value next to Disease for example, should be something like the total number of experiments selected, in response to the "Ammon's horn" filter?

Good question.
Categories used to be selectable, so the count would reflect the number of datasets if the category would be selected. Thus the count by applying every other filter (taxa, other categories, ...).
I guess we could change that behaviour to instead reflect the number of datasets in the selection. I feel however that it would be a little useless since that information is already provided by the selection.

Also, the counts are not visible in production and categories are not selectable. This is the correct behaviour.

Oh, I got confused there. The disease count does not reflect the selection in organism parts. This is either a bug or an incomplete feature. Let me check.

I looked up the code. The counts per category only reflect taxa and platform filters. Filters from other categories are simply ignored.

Since this is only apparent in the debug mode, which I didn't realize, this isn't really an issue.


A concise way to persist a GraphQL Argument in Scoped State

Is your feature request related to a problem? Please describe.
Currently I'm able to set scoped state in a Resolver in two ways:

Using the ResolverContext context.ScopedContextData.SetItem

Using SetState<T>

If I want to persist a GraphQL Argument in scoped state, I need two resolver arguments, the GraphQL Argument, and the ResolverContext or SetState.
Describe the solution you'd like
It would be really cool to have just one Resolver Argument namely the GraphQL Argument preceding by an Attribute like PersistToScopedState or something like that.

@rstaib
One problem I see with this are the linters. IDE and Sonar would see it as a issue that a parameter is not used
public class Query
{
      public Foo GetFoo([PersistToScopedState]string bar) => new Foo();
}

@PascalSenn it's for a specific use-case where you want to do both, use the argument but also store it in the scoped state.


Exits full screen when hovering over page buttons

Hi @tblanke
I noticed some strange behaviour. When the number of pages is as high as your screen height (which is the case for this specific laptop), whenever you hover over a page button, the window exits full screen.
I changed the example document in this branch that illustrates my problem. If you don't notice the escape from the full screen, perhaps your screen is higher than mine and you need to add extra pages to see this bug.
Best,
@wouterpeere

Hi @wouterpeere ,
I think I have fixed the issue. Can you please take a look?
With best regards
@tblanke

Hi @tblanke
The problem is indeed solved when hovering over the page buttons.
However, when you press calculate and the result page pops open, the window is still exitting full screen mode ...
Best regards,
@wouterpeere

Hi @WouterEnead
I Think this should be now fixed as well.
With best regards
@tblanke

Hi @tblanke
I confirm!
Best,
@wouterpeere


Blazor localization

Fixes #13436
Internal Review Topic (Localization section direct link)

For the release/3.0-preview8 staging branch.
Watch me carefully here ... I might have :boom: broken something :boom:.
Placed at the bottom, but it is cross-linked both ways to the @bind section, which is where the globalization bits live.
You suggested possibly bringing some bits over from ASP.NET Core docs on the subject. Did you have anything specific in mind? I cross-link as you suggest ... will that cover it?

Thanks ... yes we do have open redirect attack coverage. I linked it on the last commit in that warning.


I can't get the values of ToggleSwitch

My code is
this.toggleSwitch.setOnChangeListener(new ToggleSwitch.OnChangeListener() {
    @Override
    public void onToggleSwitchChanged(int toggle_position) {
        MainActivity.this.arParam = toggle_position;
    }
});

Event does not fire
public void onToggleSwitchChanged(int toggle_position) {
    MainActivity.this.arParamAngle = toggle_position;
}

I've tried your code and it works for me.
I'm in trouble figuring out what your problem is.
Can you provide more code about your problem?
Anyway, @AlecSPB I have made a fork of this library that solves many issues.
The api are almost the same as the previous one.
Try with it, it should solve your issue ;)
Let me know about it.
Check it out at https://github.com/llollox/Android-Toggle-Switch


Add support for name linking in the payload

Would there be interest in a flag to enable linking names in a payload, such as @username or @channel to notify everyone in the channel on a specific message?
It looks like it should be as simple as adding the following to the payload:
"link_names": 1

Disregard, apparently slack has this built in by using the following syntax:
<!channel> or <!username>


Binary releases?

Found that Release Archive is containing only source code. I think it will be better if there will be EXE releases of the wdd utility for known Windows architectures (x86, amd64 and ARM 32/64). Not all users have C/C++ build environment.

I have uploaded binary packages to the release page. Thank you for the suggestion.


Avro->Protobuf conversion, including fixes for the testsuite

Further work towards #784, squashed down from #799

OK, after discussion with @palfrey and others we've decided to merge this to the protobuf branch. The status is, this implements a version of the schema that is very close to the schemas proposed in https://github.com/ga4gh/schemas/pull/547, but not quite identical. Subsequent PRs against the protobuf branch will seek to fix this.
Thanks to @palfrey for the great work.


Using Square.OkHttp with Android 4.2, a TLS 1.2 site and SSL self-signed certificates

I found the square.OkHttp on Nuget and I tried to use it in Xamarin but it throws an error. Can you perhaps help?
Here is the code I am using in the MainActivity class:
StrictMode.ThreadPolicy policy = new StrictMode.ThreadPolicy.Builder().PermitAll().Build();
StrictMode.SetThreadPolicy(policy);
OkHttpClient client = new OkHttpClient();
Request request = new Request.Builder().Url("https://<yourserver>").Build();
Response response = await client.NewCall(request).ExecuteAsync();
string body = response.Body().String();
System.Diagnostics.Debug.WriteLine (body);

And the error I get is this:
[MonoDroid] UNHANDLED EXCEPTION:
[MonoDroid] Javax.Net.Ssl.SSLHandshakeException: Exception of type 'Javax.Net.Ssl.SSLHandshakeException' was thrown.
[MonoDroid] at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <IL 0x00011, 0x0004b>
[MonoDroid] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<ThrowAsync>m__0 (object) <IL 0x00006, 0x0003b>
[MonoDroid] at Android.App.SyncContext/<Post>c__AnonStorey0.<>m__0 () [0x00000] in /Users/builder/data/lanes/1879/5f55a9ef/source/monodroid/src/Mono.Android/src/Android.App/SyncContext.cs:18
[MonoDroid] at Java.Lang.Thread/RunnableImplementor.Run () [0x0000b] in /Users/builder/data/lanes/1879/5f55a9ef/source/monodroid/src/Mono.Android/src/Java.Lang/Thread.cs:36
[MonoDroid] at Java.Lang.IRunnableInvoker.n_Run (intptr,intptr) [0x00009] in /Users/builder/data/lanes/1879/5f55a9ef/source/monodroid/src/Mono.Android/platforms/android-21/src/generated/Java.Lang.IRunnable.cs:71
[MonoDroid] at (wrapper dynamic-method) object.019412e5-7311-4cc0-a58e-164d30934145 (intptr,intptr) <IL 0x00011, 0x0001f>
[MonoDroid]   --- End of managed exception stack trace ---
[MonoDroid] javax.net.ssl.SSLHandshakeException: javax.net.ssl.SSLProtocolException: SSL handshake aborted: ssl=0xb8c5b2e8: Failure in SSL library, usually a protocol error
[MonoDroid] error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version (external/openssl/ssl/s23_clnt.c:741 0x971ea990:0x00000000)
[MonoDroid] 	at com.android.org.conscrypt.OpenSSLSocketImpl.startHandshake(OpenSSLSocketImpl.java:448)
[MonoDroid] 	at com.squareup.okhttp.internal.http.SocketConnector.connectTls(SocketConnector.java:103)
[MonoDroid] 	at com.squareup.okhttp.Connection.connect(Connection.java:143)
[MonoDroid] 	at com.squareup.okhttp.Connection.connectAndSetOwner(Connection.java:185)
[MonoDroid] 	at com.squareup.okhttp.OkHttpClient$1.connectAndSetOwner(OkHttpClient.java:128)
[MonoDroid] 	at com.squareup.okhttp.internal.http.HttpEngine.nextConnection(HttpEngine.java:341)
[MonoDroid] 	at com.squareup.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:330)
[MonoDroid] 	at com.squareup.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:248)
[MonoDroid] 	at com.squareup.okhttp.Call.getResponse(Call.java:273)
[MonoDroid] 	at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:230)
[MonoDroid] 	at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:201)
[MonoDroid] 	at com.squareup.okhttp.Call.access$100(Call.java:36)
[MonoDroid] 	at com.squareup.okhttp.Call$AsyncCall.execute(Call.java:164)
[MonoDroid] 	at com.squareup.okhttp.internal.NamedRunnable.run(NamedRunnable.java:33)
[MonoDroid] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
[MonoDroid] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
[MonoDroid] 	at java.lang.Thread.run(Thread.java:841)
[MonoDroid] 	Suppressed: javax.net.ssl.SSLHandshakeException: javax.net.ssl.SSLProtocolException: SSL handshake aborted: ssl=0xb8c29508: Failure in SSL library, usually a protocol error
[MonoDroid] error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version (external/openssl/ssl/s23_clnt.c:741 0x971ea990:0x00000000)
[MonoDroid] 		... 17 more
[MonoDroid] 	Caused by: javax.net.ssl.SSLProtocolException: SSL handshake aborted: ssl=0xb8c29508: Failure in SSL library, usually a protocol error
[MonoDroid] error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version (external/openssl/ssl/s23_clnt.c:741 0x971ea990:0x00000000)
[MonoDroid] 		at com.android.org.conscrypt.NativeCrypto.SSL_do_handshake(Native Method)
[MonoDroid] 		at com.android.org.conscrypt.OpenSSLSocketImpl.startHandshake(OpenSSLSocketImpl.java:405)
[MonoDroid] 		... 16 more
[MonoDroid] Caused by: javax.net.ssl.SSLProtocolException: SSL handshake aborted: ssl=0xb8c5b2e8: Failure in SSL library, usually a protocol error
[MonoDroid] error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version (external/openssl/ssl/s23_clnt.c:741 0x971ea990:0x00000000)
[MonoDroid] 	at com.android.org.conscrypt.NativeCrypto.SSL_do_handshake(Native Method)
[MonoDroid] 	at com.android.org.conscrypt.OpenSSLSocketImpl.startHandshake(OpenSSLSocketImpl.java:405)
[MonoDroid] 	... 16 more
[AndroidRuntime] Shutting down VM

Does this fix it? https://gist.github.com/mattleibow/c8abfa323db094b820cc
In that gist, I created an wrapper socket factory that made sure to enable all the supported protocols:
sslSocket.SetEnabledProtocols (sslSocket.GetSupportedProtocols ());
Docs: http://developer.android.com/reference/javax/net/ssl/SSLSocket.html
Note, you shouldn't have to adjust the thread policy:
OkHttpClient client = new OkHttpClient();
Request request = new Request.Builder().Url("https://www.blinc.co").Build();
// UI thread here
Response response = await client.NewCall(request).ExecuteAsync(); // run on a new thread here
// UI thread here
string body = await response.Body().StringAsync(); // read on a new thread here
// UI thread here
System.Diagnostics.Debug.WriteLine (body);

There are several ways you can use async and Task to read the response:
var body = response.Body (); // no work
using (var bodyStream = body.ByteStream ())
using (var memory = new MemoryStream ()) {
    await bodyStream.CopyToAsync (memory); // read work
}
var bodyBytes = await body.BytesAsync (); // read work
var bodyString = await body.StringAsync (); // read work

Example usage of the gist in the provided example would be:
OkHttpClient client = new OkHttpClient ();
// only include our custom factory for the older Androids
if (Android.OS.Build.VERSION.SdkInt < BuildVersionCodes.L) {
	client.SetSslSocketFactory (new CompleteSSLSocketFactory ());
}
Request request = new Request.Builder ().Url ("https://www.blinc.co").Build ();
// UI thread here
Response response = await client.NewCall (request).ExecuteAsync (); // run on a new thread here
// UI thread here
string body = await response.Body ().StringAsync (); // read on a new thread here
// UI thread here
System.Diagnostics.Debug.WriteLine (body);

Thank you. This works well but introduced a new challenge. Prior to this solution I used the HttpClientHandler in the PCL project for as the http client and all I needed to do is to add the following to the MainActivity OnCreate method to allow self-signed server certificates.
ServicePointManager.ServerCertificateValidationCallback += new RemoteCertificateValidationCallback(AllwaysGoodCertificate);
And the following method:
private static bool AllwaysGoodCertificate(object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors policyErrors) {
return true;
}
With the new solution self-signed certificates don't work. Following the examples provided here
http://stackoverflow.com/questions/1217141/self-signed-ssl-acceptance-android
there seem to be two approaches:
A) Provide my own host name verifier and trust manager
B) Create a socket factory that reads from your own local key store used in the app and create an http client that registers its own scheme using the socketfactory
Any thoughts?

It really depends on what the self-signed certificates are used for.
If it is purely for development, then you could probably safely just ignore all certificates. It is highly advisable to make sure any such code is wrapped in #if DEBUG blocks to avoid any accidental deployment to a production server.
However, it is not too hard to provide the certificate in the Android app. I have updated the gist to demonstrate 2 ways in which we can handle self-signed certificates, or certificates from an unknown CA.
The gist that has been updated: http://bit.ly/1Mn90In
Here is the code that will take a .cer file from a website and crate a new keystone at runtime. This can then be used to create a trust manager that is then given to a SSL context which is where we get our SSL socket factory from:
// Load our certificate from resources (we created this one using OpenSSL and
// saved as a .cer using Windows' certlm console)
var certificateFactory = CertificateFactory.GetInstance ("X.509");
Certificate certificate;
using (var stream = Application.Context.Resources.OpenRawResource (Resource.Raw.selfsigned)) {
    certificate = certificateFactory.GenerateCertificate (stream);
}

// Create a KeyStore containing our trusted CAs
var keyStore = KeyStore.GetInstance (KeyStore.DefaultType);
keyStore.Load (null, null);
keyStore.SetCertificateEntry ("ca", certificate);

// Create a TrustManager that trusts the CAs in our KeyStore
var trustManagerFactory = TrustManagerFactory.GetInstance (TrustManagerFactory.DefaultAlgorithm);
trustManagerFactory.Init (keyStore);
var trustManager = trustManagerFactory.GetTrustManagers () [0].JavaCast<IX509TrustManager> ();

// Create an SSLContext that uses our TrustManager
var context = SSLContext.GetInstance ("TLSv1.2");
context.Init (null, new ITrustManager[]{ trustManager }, null);

// apply the new context
var socketFactory = context.SocketFactory;

It is highly advisable to not just use a single certificate as done here:
context.Init (null, new ITrustManager[]{ trustManager }, null);

Doing this will prevent all other SSL sites from working. You can use a wrapped trust manager that first tries the default trust manager, and then tries our custom trust manager. Depending on which requests are more common, they can be the other way around:
public void CheckServerTrusted (X509Certificate[] chain, string authType)
{
    try {
        defaultTrustManager.CheckServerTrusted (chain, authType);
    } catch (CertificateException) {
        localTrustManager.CheckServerTrusted (chain, authType);
    }
}

Here are some docs that are a good read: http://developer.android.com/training/articles/security-ssl.html#UnknownCa

Thank you. This was very helpful.

Matthew,
Have you seen examples out there of using self-signed client certificates?
Kind regards,
Nima

On 24 Jul 2015, at 23:48, Matthew Leibowitz<EMAIL_ADDRESS>wrote:
It really depends on what the self-signed certificates are used for.
If it is purely for development, then you could probably safely just ignore all certificates. It is highly advisable to make sure any such code is wrapped in #if DEBUG blocks to avoid any accidental deployment to a production server.
However, it is not too hard to provide the certificate in the Android app. I have updated the gist to demonstrate 2 ways in which we can handle self-signed certificates, or certificates from an unknown CA.
The gist that has been updated: http://bit.ly/1Mn90In
Here is the code that will take a .cer file from a website and crate a new keystone at runtime. This can then be used to create a trust manager that is then given to a SSL context which is where we get our SSL socket factory from:
// Load our certificate from resources (we created this one using OpenSSL and
// saved as a .cer using Windows' certlm console)
var certificateFactory = CertificateFactory.GetInstance ("X.509");
Certificate certificate;
using (var stream = Application.Context.Resources.OpenRawResource (Resource.Raw.selfsigned)) {
certificate = certificateFactory.GenerateCertificate (stream);
}
// Create a KeyStore containing our trusted CAs
var keyStore = KeyStore.GetInstance (KeyStore.DefaultType);
keyStore.Load (null, null);
keyStore.SetCertificateEntry ("ca", certificate);
// Create a TrustManager that trusts the CAs in our KeyStore
var trustManagerFactory = TrustManagerFactory.GetInstance (TrustManagerFactory.DefaultAlgorithm);
trustManagerFactory.Init (keyStore);
var trustManager = trustManagerFactory.GetTrustManagers () [0].JavaCast ();
// Create an SSLContext that uses our TrustManager
var context = SSLContext.GetInstance ("TLSv1.2");
context.Init (null, new ITrustManager[]{ trustManager }, null);
// apply the new context
var socketFactory = context.SocketFactory;
It is highly advisable to not just use a single certificate as done here:
context.Init (null, new ITrustManager[]{ trustManager }, null);
Doing this will prevent all other SSL sites from working. You can use a wrapped trust manager that first tries the default trust manager, and then tries our custom trust manager. Depending on which requests are more common, they can be the other way around:
public void CheckServerTrusted (X509Certificate[] chain, string authType)
{
try {
defaultTrustManager.CheckServerTrusted (chain, authType);
} catch (CertificateException) {
localTrustManager.CheckServerTrusted (chain, authType);
}
}
Here are some docs that are a good read: http://developer.android.com/training/articles/security-ssl.html#UnknownCa
—
Reply to this email directly or view it on GitHub.

Mazloumi
If i were you i would avoid the trouble and get a free ssl certificate from Comodo, its great for testing without the hassle.

As @josephkandi that is probably the best option, then, there is no issues at all. However, if that is not possible, you should be able to use certificates from a trusted source - as in the example of https://certs.cac.washington.edu/CAtest. I wouldn't have my server using my self-signed certificates. Especially since I many not be the only client. Also, when using self-signed certificates, any changes to the certificate will require updates to all the clients - which cannot always be guaranteed.
The only real case where self-signed certificates might be used would be in an internal, enterprise situation, where the server and the clients are carefully controlled. But then again, why not just buy a certificate and avoid any issues altogether.


Not able to work with GCC High after connecting "Location Header is not set in response"

Thanks for reporting the bug. Please ensure you've gone through the following checklist before opening an issue:

Make sure you can reproduce this issue using the latest released version of Microsoft.Graph.Entra or Microsoft.Graph.Entra.Beta.
Please search the existing issues to see if there has been a similar issue filed.

Describe the bug
After to connecting to a GCC High tenant, unable to execute any commands. They all respond with "Unable to perform redirect as Location Header is not set in response". Do NOT receive this behavior in a Commercial tenant.
To Reproduce
Steps to reproduce the behavior:

Execute Connect-Entra with -Environment USGov -Scopes Directory.AccessAsUser.All
Execute 'Get-EntraUser'

Expected behavior
No errors.
Debug Output
DEBUG: ============================ TRANSFORMATIONS ============================
DEBUG: Uri : https://graph.microsoft.com/v1.0/users/?$select=Id,AccountEnabled,AgeGroup,OfficeLocation,AssignedLicenses,AssignedPlans,City,CompanyName,ConsentProvidedForMinor,Country,CreationType,Department,DisplayName,GivenName,OnPremisesImmutableId,JobTitle,LegalAgeGroupClassification,Mail,MailNickName,MobilePhone,OnPremisesSecurityIdentifier,OtherMails,PasswordPolicies,PasswordProfile,PostalCode,PreferredLanguage,ProvisionedPlans,OnPremisesProvisioningErrors,ProxyAddresses,RefreshTokensValidFromDateTime,ShowInAddressList,State,StreetAddress,Surname,BusinessPhones,UsageLocation,UserPrincipalName,ExternalUserState,ExternalUserStateChangeDateTime,UserType,OnPremisesLastSyncDateTime,ImAddresses,SecurityIdentifier,OnPremisesUserPrincipalName,ServiceProvisioningErrors,IsResourceAccount,OnPremisesExtensionAttributes,DeletedDateTime,OnPremisesSyncEnabled,EmployeeType,EmployeeHireDate,CreatedDateTime,EmployeeOrgData,preferredDataLocation,Identities,onPremisesSamAccountName,EmployeeId,EmployeeLeaveDateTime,AuthorizationInfo,FaxNumber,OnPremisesDistinguishedName,OnPremisesDomainName,IsLicenseReconciliationNeeded,signInSessionsValidFromDateTime
DEBUG: Method : GET
DEBUG: =========================================================================
DEBUG: GET /v1.0/users/?$select=Id,AccountEnabled,AgeGroup,OfficeLocation,AssignedLicenses,AssignedPlans,City,CompanyName,ConsentProvidedForMinor,Country,CreationType,Department,DisplayName,GivenName,OnPremisesImmutableId,JobTitle,LegalAgeGroupClassification,Mail,MailNickName,MobilePhone,OnPremisesSecurityIdentifier,OtherMails,PasswordPolicies,PasswordProfile,PostalCode,PreferredLanguage,ProvisionedPlans,OnPremisesProvisioningErrors,ProxyAddresses,RefreshTokensValidFromDateTime,ShowInAddressList,State,StreetAddress,Surname,BusinessPhones,UsageLocation,UserPrincipalName,ExternalUserState,ExternalUserStateChangeDateTime,UserType,OnPremisesLastSyncDateTime,ImAddresses,SecurityIdentifier,OnPremisesUserPrincipalName,ServiceProvisioningErrors,IsResourceAccount,OnPremisesExtensionAttributes,DeletedDateTime,OnPremisesSyncEnabled,EmployeeType,EmployeeHireDate,CreatedDateTime,EmployeeOrgData,preferredDataLocation,Identities,onPremisesSamAccountName,EmployeeId,EmployeeLeaveDateTime,AuthorizationInfo,FaxNumber,OnPremisesDistinguishedName,OnPremisesDomainName,IsLicenseReconciliationNeeded,signInSessionsValidFromDateTime HTTP/1.1
HTTP: graph.microsoft.com
User-Agent: PowerShell/7.4.6 EntraPowershell/0.19.0 Get-EntraUser
Invoke-MgGraphRequest: Unable to perform redirect as Location Header is not set in response
Module Version
0.19.0
Environment Data
Name                           Value

PSVersion                      7.4.6
PSEdition                      Core
GitCommitId                    7.4.6
OS                             Microsoft Windows 10.0.26100
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0
Screenshots

Additional context

Thank you, @ryan-indirectit, for bringing this to our attention. We've reproduced the issue and identified outdated dependencies as the likely root cause. We’ll update the MS Graph PowerShell SDK dependencies as part of the fix in the next release and will keep you informed on our progress.


Output Class Labels

After running manyclassify.py, it gives me top 3 predicitions for all the images. But I don't understand the class labels that it assigns for example, for image 28.png the output class lables returned are14, 8, 17. Where are these labels defined and what do they mean? How can I change the target class for the stop sign?

I just added mapping.txt under lisa-cnn-attack in commit 3810af06bb0c93b4da94f0aa3e066b5f8d0cd8d0 You can confirm if it is correct by running clean images through the network and comparing the expected label to what the network spits out. Because it has been a while and I might have copied the wrong file, they might be one off; if so, please, feel free to reopen and I will look for the correct file.

Hey, thanks for your help and for providing the list. I have seen the text file but there is the Issue that the model always classifies the images to the "Class 14"  with the highest probability and "Class 14" belongs to "Stop Sign" which means it classifies stop sign as stop sign whereas it should actually misclassify the stop sign as something else.
Also, I would like to know how to set a target class of my choosing so that the network will always return the class of my choice?


explicit additional steps to integrate Pickles & SpecFlow report

As promised on https://github.com/picklesdoc/pickles/issues/460

Thank you very much!


Use updated URL in about section

Describe your use-case which is not covered by existing documentation.
https://plugins.jenkins.io/coverage-badges-extension/ use this URL instead of the older GitHub URL in the about section of the plugin GH page
Reference any relevant documentation, other materials or issues/pull requests that can be used for inspiration.
No response
Are you interested in contributing to the documentation?
No response

Thanks, I've updated the link


work on song routes using spotify api

AS A USER
WHEN i search for songs through spotify api
A LIST of songs are displayed for me to add to the event i want to add it to

this is working for the most part. few bugs i will create separate issues for the bug fixes


Calling toString() on potentially incomplete buffers

It seems byline is just calling toString() on buffers without correctly respecting unicode encoding rules. Since the buffers going into transform can be on any arbitrary position inside the original byte stream it is possible for it to be in the middle of a character.
This would potentially influence all characters encoded with more than 1 bytes.
if (encoding == 'buffer') {
  chunk = chunk.toString(); // utf8
  encoding = 'utf8';
}

Is this handled somewhere else ?

i just got an error like that with some Chinese characters where I didn't get the whole json that is present as expected, @matthiasg do you know how can I correct this issue?

@LeonFedotov you could use the stringdecoder as built into node directly. it can be continuously fed with new bytes and will emit proper unicode parsed strings only. each string can then be fed into byline (or just do it yourself).

As a workaround, for file streams, you can set the source stream encoding:
const stream = byline(fs.createReadStream(filename, {encoding: 'utf-8'}))


python312Packages.markdown: 3.6 -> 3.7

https://github.com/Python-Markdown/markdown/blob/refs/tags/3.7/docs/changelog.md
Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

#335329


package-lock.json is not updated if run with npx

My use case: I don't have npmversion installed in my packages nor expressed in scripts.
Instead I use it in a CI job which purpose is to do a version bump and a release.
One of my job steps is npm i --no-save npmversion && node ./node_modules/.bin/npmversion --increment ${INCREMENT} --preid ${PREID}
I tried to replace it with npx npmversion --increment ${INCREMENT} --preid ${PREID} which theoretically should do the same in one neat expression.
But it doesn't.
The "manual" approach results in 2 files changed: package.json and package-lock.json.
The npx approach results in only package.json being changed unless package-lock.json is explicitly specified in jsonFiles in .npmversionrc

Hi
To achieve that, I run the command "npm version" which does the job to change the  project verson in package.json, package-lock.json and npm-shrinkwrap.json. So even if the command is running with npx, internally, we run npm.
Have you check if there are some issues around calling npm command (like version) through npx ?
I try to have a look on my side
Regards

Weird because running the following command:

node ./node_modules/npmversion/bin/npmversion --increment patch

It works fine: the package.json and package-lock.json are well updated
Now if I run:

npx npmversion --increment patch

Both of the package.json and package-lock.json are updated BUT I have an error around the npm error-handler "npm/lib/utils/error-handler.js:205:18" for the evaluation of "if (npm.config.get('json')) {"
It seems however that the process works and don't prevent to bump and create the commit
My context:

Windows 10 x64
Node v8.11.1
NPM 5.6.0
NPMVERSION 1.70
NPX 9.7.1

I try to check that on a linux
Regards

@rochejul I checked on linux Jenkins machine and on a Mac
Node 8.9.1 on mac and 8.11.x on Jenkins
NPM 5.5.1 on mac, 5.6.0 on Jenkins

Ok I try to take some time on the next days to check on my ubuntu (and with nvm) if there is a specific version with an error
Many thanks for your feedback
Regards

Seems to be linked to https://github.com/zkat/npx/issues/143 and to https://github.com/zkat/npx/issues/100

Ok, on my case (for windows), my profile name contains a space (sigh) which produces an error according to https://github.com/zkat/npx/issues/100
Now I have to check why in your case you don't have the expected behavior
Regards

Hummmm very weird because on my ubuntu:

And I have the following configuration:

node 8.9.1
npm 5.5.1
npmversion 1.7.0
npx 10.2.0

What is your npx version ?
Many thanks

9.6.0 on mac, but I updated and it's the same
➜  test-npmversion-npx git:(master) npx --version
9.6.0
➜  test-npmversion-npx git:(master) npm i -g npx
/Users/jrencz/.nvm/versions/node/v8.9.1/bin/npx -> /Users/jrencz/.nvm/versions/node/v8.9.1/lib/node_modules/npx/index.js
+<EMAIL_ADDRESS>added 484 packages in 6.423s
➜  test-npmversion-npx git:(master) npx npmversion --increment prepatch --preid npxtest --nogit-commit --nogit-tag
v0.0.3-npxtest.0
➜  pl-fe-ui-angularjs git:(master) ✗ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   package.json

no changes added to commit (use "git add" and/or "git commit -a")

9.6.0 on mac, but I updated and it's the same
➜  test-npmversion-npx git:(master) npx --version
9.6.0
➜  test-npmversion-npx git:(master) npm i -g npx
/Users/jrencz/.nvm/versions/node/v8.9.1/bin/npx -> /Users/jrencz/.nvm/versions/node/v8.9.1/lib/node_modules/npx/index.js
+<EMAIL_ADDRESS>added 484 packages in 6.423s
➜  test-npmversion-npx git:(master) npx npmversion --increment prepatch --preid npxtest --nogit-commit --nogit-tag
v0.0.3-npxtest.0
➜  test-npmversion-npx git:(master) ✗ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   package.json

no changes added to commit (use "git add" and/or "git commit -a")

I checked one more thing: after analysing npmversion code I found that it's not really npmversion who adjusts the version in package.json and package-lock.json unless it's directly instructed to via jsonFiles.
I run npm --no-git-tag-version version 0.0.4 and it updated both files
But then I run npx npm --no-git-tag-version version 0.0.4 and... it adjusted just package.json
So I checked with latest node, npm and npx:
➜  text-npmversion-npx git:(master) ✗ nvm i 10.8
######################################################################## 100.0%
Checksums empty
Now using node v10.8.0
➜  text-npmversion-npx git:(master) ✗ npm --version
6.2.0
➜  text-npmversion-npx git:(master) ✗ npm i -g npm
/Users/jrencz/.nvm/versions/node/v10.8.0/bin/npm -> /Users/jrencz/.nvm/versions/node/v10.8.0/lib/node_modules/npm/bin/npm-cli.js
/Users/jrencz/.nvm/versions/node/v10.8.0/bin/npx -> /Users/jrencz/.nvm/versions/node/v10.8.0/lib/node_modules/npm/bin/npx-cli.js
+<EMAIL_ADDRESS>added 1 package from 1 contributor and updated 3 packages in 5.869s
➜  text-npmversion-npx git:(master) ✗ npx --version
6.3.0
➜  text-npmversion-npx git:(master) ✗ npm i -g npx
/Users/jrencz/.nvm/versions/node/v10.8.0/bin/npx -> /Users/jrencz/.nvm/versions/node/v10.8.0/lib/node_modules/npx/index.js
+<EMAIL_ADDRESS>added 484 packages from 651 contributors in 5.833s
➜  text-npmversion-npx git:(master) ✗ npx npm --no-git-tag-version version 0.0.4
v0.0.4
➜  text-npmversion-npx git:(master) ✗ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   package.json

no changes added to commit (use "git add" and/or "git commit -a")
➜  text-npmversion-npx git:(master) ✗

I'm confused

I repeated npx npm --no-git-tag-version version 0.0.4 on a APFS partition (just default, case insensitive) filesystem and it worked.
I normally work on a case-sensitive Mac OS Extended - there the bug is reproducible.
I guess it's npx bug.
But why does it happen on linux where it's quite normal to have case sensitive filesystems?

@jrencz Yes, npmversion used "npm version" to update package.json, package-lock.json and npm-shrinkwrap.json. So if with npx, it does not work, this is probably an issue on npx side.
I saw some issue around spaces and case sensitives: zkat/npx#100
I keep this issue open until a solution could be found
regards


⚠ MorethanTV has degraded performance

In 44c26f3, MorethanTV ($MTV) experienced degraded performance:

HTTP code: 502
Response time: 4400 ms

Resolved: MorethanTV performance has improved in 4b2b3c3 after 58 minutes.


Update to latest wgpu

These are the changes necessary to update to the upcoming wgpu.
Still need to actually release wgpu-native and wgpy-py, and pin them in pygfx.

Indeed! Though it took me a while to figure out what was needed for the depth-stencil issue ...


[RFC] Promoting keys from the disk to the heap tier in TieredSpilloverCache

Is your feature request related to a problem? Please describe
Currently in tiered/pluggable caching, keys move to the on-heap tier to the disk tier when evicted by the heap tier’s LRU policy. But, unless the key is evicted from the disk tier and then re-added to the heap tier, there’s no way for keys to move back up.
Consider a key that gets hits moderately often. It can get evicted from a small LRU heap tier, since the heap tier can’t hold many keys. But, it’s not likely to be evicted from the large LRU disk tier: there are so many more keys there that it will rarely be the least-recently used key. So, all future hits on this key will come from the disk tier, which is slower than the heap tier. This problem should worsen the longer the node is running as more and more keys get stuck.
In some cases, key invalidation helps by removing stale keys from both tiers. But not always - for example, on an index of past logs, there won’t be any invalidation.
Ideally, we would keep the few most frequently-used keys in the heap tier and use the larger disk tier for a larger number of less-frequently used keys.
Describe the solution you'd like
We want the TieredSpilloverCache to keep track of how often keys get hits from the disk tier. If a key gets more hits than some threshold, we move it back to the heap tier (“promoting” it). This causes the heap tier to evict its least-recently used key into the disk tier. In practice we can’t do exactly this for performance reasons.
Keeping an in-memory hit counter for every on-disk key is not feasible. Instead, we can use a memory-efficient frequency approximating data structure like CountMinSketch, which is used by high-performance caching libraries like Caffeine. The data structure is pretty simple so we could implement it ourselves, or we could use some library’s implementation. Because the values in CountMinSketch can decay over time in a configurable way we don’t have to worry about removing keys from the structure, or about the frequency estimations creeping up over time and leading to too many promotions.
Promotions will have some small performance cost, so we can’t use small threshold values like 1. I don’t expect the cost to be very large though, as it’s just invalidating the key from the disk tier, adding it to the heap tier, and having the heap tier evict some other key. The last 2 steps already happen every time there’s a cache miss, and the invalidation can happen regularly from stale keys. So, as long as the number of promotions is << the number of total cache misses, I don’t expect much overhead from the promotions themselves.
Related component
Storage:Performance
Describe alternatives you've considered
Promoting based on just a threshold may cause significantly different numbers of promotions based on the workload, especially the fraction of queries that can be repeated. We may want to tweak this based on cache hit ratio or something similar. This would probably be tested through benchmarks.
As a more distinct alternative, we can consider using Caffeine as the heap tier instead of the LRU OpenSearch cache. Caffeine has a smarter eviction policy that approximates LFU, so the keys that go to the disk tier are less likely to get many hits in the future. This reduces the “stuck keys” problem without actually having to do any promotions. Some work has already been done on this, but we haven’t gotten it to be more performant than the default cache as of yet. It could also make sense for Caffeine to go alongside key promotion.
Additional context
Some preliminary numbers which give an idea of how big the problem is and upper bounds for how much we can expect this to improve performance:
How often do keys get stuck in the disk tier?
We can run a simulation of the randomized nyc_taxis workload which we’ve used to measure TC performance, where we track frequencies of hits for all keys in both heap and disk tier. Then we can see if keys actually do get stuck. We can measure the fraction of total hits coming from the heap tier, or “heap hit rate fraction” / “HHRF”. Maximizing HHRF should increase performance.

The keys are ordered along the x-axis by their “Zipf index” i. In each benchmark iteration, this determines the probability we pick the query which matches key i, where smaller values of i are the most probable. Key i has probability proportional to 1 / i^alpha. This follows how randomization works in OSB. Note the log-scale x-axis. This figure is set up so the area of orange/blue is proportional to the total number of heap/disk hits.
Orange means a hit from the heap tier; blue means a hit from the disk tier. We can see the few most common keys are always served from the heap tier (their whole bar is orange). But, as early as the ~10th most common key, we see hits from the disk tier. For cheap queries this happens by the ~30th most common key; cheap queries happen more often in the test so their keys are less likely to be evicted from the heap tier.
How much does enabling promotion increase HHRF?
In the simulation, we increment a hit counter every time a disk-tier key gets a hit. If this counter reaches some threshold value we immediately move it up to the heap tier. This is an ideal case of what we might do in practice.
If we run the above test with various values for this threshold, we see varying levels of improvement:

The most improvement in HHRF is seen for small threshold values, with the minimum value of 1 performing the best by far. But even for large values like 20, we see substantial improvement over the baseline with no promotion at all.
The number of promotions was much higher for small threshold values:

But even for large values with few promotions, we still see a significant HHRF improvement. Of course this could be pretty workload-dependent.

[Catch All Triage - 1, 2, 3]


Enable service account impersonation in GoogleCalendar source connector

This enable service account impersonation (in case the user is organization manager & want to use a service account to access memebers' calendars)
Description
Current implementation of GoogleCalendar source connector only allows calendar owner or invited accounts to access calendar.
In normal case, this is fine. But in an organization, when admins need to access members' calendars to calculate necessary metrics to determine productivity, forcing all members to share their calendar with an account is tricky.
This PR enable service account impersonation (together with domain-wide delegation), so that an authenticated service account can access organization members' calendars.
Type of change

[ ] Bug fix
[x] New feature
[ ] Breaking change

I have read the CLA Document and I hereby sign the CLA

@tovbinm can you have a look at this?

FYI @lhvubtqn https://github.com/faros-ai/airbyte-connectors/pull/792


UI: wrap all text for translation

This wraps all strings in the Qt tr function, which translates the source text using the context (parent class) and any disambiguation text. If no translations are available it's a noop and returns the text passed in. This is to remove some diff from the main multilang PR: https://github.com/commaai/openpilot/pull/21212
For my reference later: doc about contexts: https://wiki.qt.io/QtInternationalization
Test goes through all QLabel and QPushButtons and makes sure that if the widget's text is not empty, it contains our testing string (which means Qt was able to translate it using the .ts translation file).

Hmm, if we intend to do dynamic translations in the future, perhaps we don't need to rely on any Qt tools for this, including the tr function

@deanlee any idea why test_translation refuses to run, even manually? I noticed test_sound also isn't running (as far as I know, I don't see any references to it).

I don't know why, but I'll check into it.

Thanks!

@deanlee ah nevermind about this PR's issues. Do you want to open a PR to run the test_sound test though?

@sshane There is no sound card on the server, it may not be possible run test_sound in selfdrive_tests ( https://github.com/commaai/openpilot/pull/24990 ) .

I'm almost 100% sure those are just for QT Linguist to show the source code to the translator. You can specify a context without a line number and it translates.

Sounds like a plan. I don't think it's necessary for this PR, as this is just to test all strings are wrapped, and thus the file is cleared every time. I'll add it to the main PR/the PR that adds a language.


[Secure Storage] Added missing README to secure storage directory.

Motivation
This PR adds the missing README file to the secure storage directory. This file gives a brief overview of secure storage and should help readers better understand the role secure storage plays in our ecosystem.
Have you read the Contributing Guidelines on pull requests?
Yes.
Test Plan
Viewed the file locally in markdown and everything appears to render correctly.
Related PRs
None.

Thanks, David! I've removed the overly explicit information from the README 😄

/libra

:exclamation: Invalid command

Bors help and documentation

Bors is a Github App used to manage merging of PRs in order to ensure that CI is always green. It does so by maintaining a Merge Queue. Once a PR reaches the head of the Merge Queue it is rebased on top of the latest version of the PR's base-branch (generally master) and then triggers CI. If CI comes back green the PR is then merged into the base-branch. Regardless of the outcome, the next PR is the queue is then processed.
General

This project's Merge Queue can be found here.
This project requires PRs to be Reviewed and Approved before they can be queued for merging.
Before PR's can be merged they must be configured with "Allow edits from maintainers" enabled. This is needed for Bors to be able to update PR's in-place so that Github can properly recognize and mark them as "merged" once they've been merged into the upstream branch.

Commands
Bors actions can be triggered by posting a comment which includes a line of the form /<action>.

Command
Action
Description

Land
land, merge
attempt to land or merge a PR

Canary
canary, try
canary a PR by performing all checks without merging

Cancel
cancel, stop
stop an in-progress land

Cherry Pick
cherry-pick <target>
cherry-pick a PR into <target> branch

Priority
priority
set the priority level for a PR (high, normal, low)

Help
help, h
show this help message

Options
Options for Pull Requests are configured through the application of labels.

          Option
Description

Indicates that the PR is high-priority. When queued the PR will be placed at the head of the merge queue.

Indicates that the PR is low-priority. When queued the PR will be placed at the back of the merge queue.

Before merging the PR will be squashed down to a single commit, only retaining the commit message of the first commit in the PR.

/land

:exclamation: Land has been canceled due to this PR being updated with new commits. Please issue another Land command if you want to requeue this PR.

/land

/land

@JoshLind :bulb: This PR is already queued for landing

:broken_heart: Test Failed - commit-workflow

/land

:broken_heart: Test Failed - commit-workflow

/land


[SPARK-20923] turn tracking of TaskMetrics._updatedBlockStatuses off

What changes were proposed in this pull request?
Turn tracking of TaskMetrics._updatedBlockStatuses off by default. As far as I can see its not used by anything and it uses a lot of memory when caching and processing a lot of blocks.  In my case it was taking 5GB of a 10GB heap and I even went up to 50GB heap and the job still ran out of memory.  With this change in place the same job easily runs in less then 10GB of heap.
We leave the api there as well as a config to turn it back on just in case anyone is using it.
How was this patch tested?
Ran unit tests that were modified and manually tested on a couple of jobs (with and without caching).  Clicked through the UI and didn't see anything missing.
Ran my very large hive query job with 200,000 small tasks, 1000 executors, cached 6+TB of data this runs fine now whereas without this change it would go into full gcs and eventually die.

Test build #77594 has started for PR 18162 at commit 5327ba5.

@JoshRosen

Test build #77594 has finished for PR 18162 at commit 5327ba5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77594/
Test PASSed.

@cloud-fan has a good point about backwards-compatibility. In case any folks are actually relying on this behavior, I wonder whether we could mark it as deprecated and have a flag for disabling it instead of complete removal.

Updated, I put the TaskMetrics api back with deprecated marking and just had it return Nil.  @JoshRosen  Were you thinking of adding more back?

Test build #77603 has started for PR 18162 at commit 41ed774.

@tgravescs, I guess the question is whether any user has a SparkListener which actually uses the value of updatedBlockStatuses for some monitoring application or something similar. In that case returning Nil will preserve binary compatibility but will break semantics for their app. I don't know of any use-cases like this offhand so I don't have a personal stake in this, but I could imagine problems in case someone relies on the return value (hence suggestion of flagging if we want to be really conservative).

taskMetrics doesn't take the sparkconf or anything to get at a config so we would have to config out everywhere its incrementing or adding things.  I think that wouldn't be to hard.  I'll put everything back and just add config around them with default off.

Test build #77603 has finished for PR 18162 at commit 41ed774.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77603/
Test PASSed.

Test build #77625 has started for PR 18162 at commit ec3c29d.

I do love the clean up you did by removing updatedBlockStatuses entirely... Since SparkListenerTaskEnd is marked as a developer API, is it acceptable to make TaskMetrics.updatedBlockStatuses always return Nil in Spark 2.3? Maybe we can send an email to dev list to ask about this.

Test build #77625 has finished for PR 18162 at commit ec3c29d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77625/
Test PASSed.

Yeah I was figuring I would file another jira to remove it later.  I can add the deprecated flag here if you guys agree.

It would be nice to get this into spark 2.2 if we can

@JoshRosen what do you think should we add the deprecated?

Yea I think we should merge this to 2.2, but we need to change the default value of the new config to true, to not surprise users.

ok, I'll update the default.

Test build #77682 has started for PR 18162 at commit fbe30cf.

turned on by default for backwards compatibility but don't really agree with it.  We should make it more stable/usable for people by turning it off. I'm assuming anyone that is using this would be very minority of people, but to be safe we can leave it on and I'll file a separate jira to turn off and deprecate.  Note I'm out of office next week so if comments I might not respond for a bit.

Test build #77682 has finished for PR 18162 at commit fbe30cf.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77682/
Test FAILed.

Test build #77684 has started for PR 18162 at commit 018955a.

I'm wondering if it's doable to stop tracking the updatedBlockStatus according to a config... there are many places that update updatedBlockStatus right?

I'm not sure what you mean by its not doable?  what places are you seeing update the block statuses that I haven't covered here?  most of it was done by the BlockManager. Maybe I'm missing something in my intellij search or greps but I dont' think so.  Look to see where incUpdatedBlockStatuses, setUpdatedBlockStatuses (2 versions) are used.

Test build #77687 has started for PR 18162 at commit a875aac.

Test build #77684 has finished for PR 18162 at commit 018955a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77684/
Test FAILed.

Merged build finished. Test FAILed.

Test build #77687 has finished for PR 18162 at commit a875aac.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77687/
Test FAILed.

@tgravescs, I deployed this to our production environment (based on 2.0.0) a few days ago and haven't hit any problems with it. I think this is good to go, unless something has been added recently that uses the block statuses.
+1

sorry been out on vacation and probably won't have time this week to respond much but will update early next week. Thanks @rdblue .  I am running this in our production as well and can clearly see memory savings.

Jenkins, test this please

Test build #78300 has started for PR 18162 at commit a875aac.

Test build #78300 has finished for PR 18162 at commit a875aac.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78300/
Test FAILed.

@tgravescs can you address https://github.com/apache/spark/pull/18162#discussion_r122363701 ?

sorry missed that you had commented, yes we can change that

Test build #78460 has started for PR 18162 at commit c9bf058.

will upmerge shortly, since there are conflicts

upmerged to master and updated default and removed unneeded changes.

Test build #78466 has started for PR 18162 at commit 68fbd9f.

Test build #78466 has finished for PR 18162 at commit 68fbd9f.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78466/
Test FAILed.

failure is from previous push of code.

Jenkins, test this please

Test build #78467 has started for PR 18162 at commit 68fbd9f.

Test build #78467 has finished for PR 18162 at commit 68fbd9f.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78467/
Test FAILed.

Test build #78468 has started for PR 18162 at commit 289e993.

Test build #78460 has finished for PR 18162 at commit c9bf058.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78460/
Test PASSed.

Jenkins, test this please

Test build #78468 has finished for PR 18162 at commit 289e993.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78468/
Test PASSed.

thanks, merging to master!

thanks for the reviews @cloud-fan


turkish translation of main README.md

This is great tutorial. I translated main README of this curriculum. Thank you!
Btw happy "Hacktoberfest"

@devisasari please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
@microsoft-github-policy-service agree [company="{your company}"]

Options:

(default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
(when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

Contributor License Agreement

@microsoft-github-policy-service agree

Don't do that Mr. Bot, thank you!


Doesn't support linux/arm64

Describe the bug
It's giving me the error image with reference docker.io/kritishdhaubanjar/dynamodb-dashboard:latest was found but does not match the specified platform: wanted linux/arm64, actual: linux/amd64
To Reproduce
Steps to reproduce the behavior:

Build using M series chip for Mac

Expected behavior

Should support arm chip

Screenshots

The Docker image builds just fine on Apple Silicon, so you can probably implement this easily with a GitHub Action without needing to bother setting up a Mac.


Changes required for Sample/Mock Payload Generation Feature

This PR is a temporary fix for adding Swagger-Inflector JAR file to lib folder until we add it as an orbit.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

 All committers have signed the CLA.


Blank screen after booting Windows 11 in UTM

I have logged into this particular VM many times in the past but on starting it today I experienced a blank boot screen after the UTM logo.Although I am able to hear the Windows 11 boot sound so the issue is only display not showing in VM.
And sometimes the VM goes into recovery and asks to restart the system.
UTM 4.5.3
MacOS 15
M3 Air

VM Config attached below
config.plist.zip

I have this same issue, and have tried version beta 4.6.1, with no joy.  I think it relates to Windows 11 24H2

Looks similar to #6872

Getting the same black screen after installing a fresh ISO from CrystalFetch and following the UTM Windows 11 Guide

Duplicate of #6332


Add new funcionality

Add an option to disable the custom title bar(m_pseudoCaption) and show the OS title bar, disappearing certain widgets from the custom title bar. This to be more similar to the functionality that programs like Firefox or Chromium have.
Also a switch statement (improves performance)

Thanks!


Add spamFilter middleware to discourage abusive traffic

This adds a mechanism to discourage abusive/unwanted traffic to the public https://httpbingo.org instance.  The first attempt here will sleep for 5 seconds before responding with a 500 Internal Server Error response, but we may need to change this strategy or adopt new strategies in the future.
Fixes #1.  See that issue for more context around the current traffic we want to discourage.

Eh, unsurprisingly, sleeping for 5 seconds per spammy request just ties up too many resources at the fly.io infrastructure layer:

2023-08-30T16:38:11Z proxy[4d891692b73287] iad [warn]Instance 4d891692b73287 reached hard limit of 50 concurrent connections. This usually indicates your app is not closing connections properly or is not closing them fast enough for the traffic levels it is handling. Scaling resources, number of instances or increasing your hard limit might help.


In order to analyze the effect of advertising promotion, we will analyze:

Daily clicks and impressions
User registrations of one app
UV, seven-day retention
Multi-dimensional analysis, such as the distribution of people using different mobile phones

We use Druid to analyze，Druid is an excellent OLAP databas，most of the metrics such as daily clicks, registration, and UV are no problem
But the  seven-day retention or multi-dimensional analysis metrics,  it is difficult to analysis.
So I want to write a Druid plug-in,  using bitmap to analysis user seven-day retention and other multi-dimensional.
The feature:

Metric: bitmap，store different user or device id(int type)
Post-aggregator：bitmap‘s cardinality
Post-aggregator：operation and between bitmaps
Post-aggregator：operation or between bitmaps
Post-aggregator：operation xor between bitmaps

Good proposal. Two questions:

Is this bitmap going to support rollup during ingestion?
Is roaringbitmap is used as the underlying data structure for this bitmap?

Support Data ingestion such as kafka
Use roaringbitmap,  roaringbitmap deserialize and serialize fast, less storage space

Calculating cardinality in short time on large volume data is challengeable. We had similar implementation based on roaringbitmap too, but the problem is even the serialization & deserialization of roaringbitmap is fast enough, they're still bottleneck when calculating high cardinality data.
Another problem is during query there will be too many bitmap object creation which will put heavy load on garbage collection.
So, I'm more care about the performance of such implementation. Do you have any reports that can be shared with us?


bug(codegen): missing , when generating type parameters with jsx

Playground link
input:
const genericFn = <T,>(foo: T) => {
	type Bar = {};
};

output:
const genericFn = <T>(foo: T) => {
	type Bar = {};
};

expected output:
const genericFn = <T,>(foo: T) => {
	type Bar = {};
};

Please excuse my lack of TypeScript knowledge, but does the trailing comma in <T,> have any semantic meaning?

It breaks for .tsx.

iirc, the parser will interpret <T> as an opening jsx tag and not as a type parameter.
it's fine in non-jsx files because <T> cant be an opening jsx element.