Loading data from ManifestArrays without saving references to disk first

I am working on a feature in virtualizarr to read dmrpp metadata files and create a virtual xr.Dataset containing manifest array's that can then be virtualized. This is the current workflow:
vdatasets = parser.parse(dmrs)
# vdatasets are xr.Datasets containing ManifestArray's
mds = xr.combine_nested(list(vdatasets), **xr_combine_kwargs)
mds.virtualize.to_kerchunk(filepath=outfile, format=outformat)
ds = xr.open_dataset(outfile, engine="virtualizarr", ...)
ds.time.values

However the chunk manifest, encoding, attrs, etc. is already in mds so is it possible to read data directly from this dataset? My understanding is that once the "chunk manifest" ZEP is approved and the zarr-python reader in xarray is updated this should be possible. The xarray reader for kerchunk can accept a file or the reference json object directly from kerchunk SingleHdf5ToZarr  and MultiZarrToZarr. So similarly can we extract the refs from mds and pass it to xr.open_dataset() directly?
There probably still needs to be a function that extracts the refs so that xarray can make a new Dataset object with all the indexes, cf_time handling, and open_dataset checks.
mds = xr.combine_nested(list(vdatasets), **xr_combine_kwargs)
refs = mds.virtualize()
ds = xr.open_dataset(refs, engine="virtualizarr", ...)

Even reading directly from the ManifestArray dataset is possible but not sure how the new dataset object with numpy arrays and indexes would be separate from the original dataset
mds = xr.combine_nested(list(vdatasets), **xr_combine_kwargs)
mds.time.values

(One subtlety I'm not sure about here would be around indexes. I think you would probably want to have a solution for loading indexes as laid out in https://github.com/zarr-developers/VirtualiZarr/issues/18, and then have the indexes understand how they can be loaded.)

Another subtlety to consider is when should the CF decoding happen? You would then have effectively done open_dataset in a very roundabout way, and we need to make sure not to forget the CF decoding step in there somewhere.

Whilst I've gone off the idea of an xarray backend that loads data by default (https://github.com/zarr-developers/VirtualiZarr/issues/221), there's another potential use case for this idea: loadable_variables.
Currently there are xarray backends and virtualizarr virtual backends. But the latter has to call the former if it wants to support the loadable_variables kwarg. For example, @sharkinsspatial 's new HDF virtual reader internally calls https://github.com/zarr-developers/VirtualiZarr/blob/1dbd119993a2f900142ee8af0ab34057f28c3831/virtualizarr/readers/hdf/hdf.py#L71
which calls xr.open_dataset
https://github.com/zarr-developers/VirtualiZarr/blob/1dbd119993a2f900142ee8af0ab34057f28c3831/virtualizarr/readers/common.py#L50
which for a netCDF4 file would call xarray's netCDF4 backend to load the loadable_variables.
So to support virtualizing one filetype, we need both a virtualizarr backend and a corresponding xarray backend. For filetypes that already have xarray backends this isn't a problem, but if we want 3rd parties to add their own virtual backends for custom filetypes, it's doubling the work they have to do.

@ayushnag I feel like there might be some sneaky trick we could do to load the ManifestArrays into memory using xr.open_zarr using either xpublish or some kind of special python zarr store implementation...
Like what if we create some kind of special ManifestStore class along the lines of all the zarr stores listed here, then use the zarr-python library to read from that? Is that possible?
https://zarr.readthedocs.io/en/stable/api/storage.html

Perhaps a Store very similar to an IcechunkStore except with a subset of the capabilities and requirements? It can have just the portion that provides the abstraction to read from a remote store given only the manifest

Yes exactly! And instead of finding the byte range info from files on disk it just gets them from the ManifestArray that's in memory.
The advantage of this approach is that we should be able to then just call open_zarr internally to get at the loadable variables, guaranteeing that Xarray will handle CF decoding of those variables in the way it currently does when we use the netCDF backend.

@ayushnag following your suggestion on slack, maybe what we need is to temporarily create an  icechunkstore then read from it? Though that would introduce a dependency on icechunk outside of the actual .to_icechunk method.
Alternatively maybe creating an FsspecStore (or whatever it's called exactly) that can read bytes from S3 ends up being functionally equivalent to your suggestion to use kerchunk above.
It would be cool to find a way to do this using just zarr-python.
cc @mpiannucci and @jhamman for feedback on this idea


Add Helm 0.3 template renderer

This is a small commit that just adds the renderer for the Helm 0.3 engine. I'm not sure how best to integrate it into expandybird, but I figured this would grease the skids.
This depends on #195.

Last commit LGTM, but can't merge until #195 is addressed.

Closing this and merging the code into feat/chartfile.


Envoy listener drain(NETWORK_FILTER change) should not cause data plane impact

We observed whenever apply below configure EnvoyFilter, it will cause listener drain in both ingressgateway and sidecar and will result to massive connection reset. Ideally xDS change should not cause data plane impact, is this expected behavior or an Envoy feature gap/bug?
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  creationTimestamp: "2023-08-10T21:09:34Z"
  generation: 2
  name: sample-200-response-code-accesslog-1.20
  namespace: istio-production
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: ANY
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
      proxy:
        proxyVersion: ^1\.20.*
    patch:
      operation: MERGE
      value:
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          access_log:
          - filter:
              and_filter:
                filters:
                - not_health_check_filter: {}
                - or_filter:
                    filters:
                    - and_filter:
                        filters:
                        - runtime_filter:
                            percent_sampled:
                              denominator: HUNDRED
                              numerator: 1
                            runtime_key: http_ok_response_sampling_fraction
                            use_independent_randomness: true
                        - status_code_filter:
                            comparison:
                              op: EQ
                              value:
                                default_value: 200
                                runtime_key: http_ok_response_sampling_status_eq
                    - and_filter:
                        filters:
                        - runtime_filter:
                            percent_sampled:
                              denominator: HUNDRED
                              numerator: 100
                            runtime_key: http_err_response_sampling_fraction
                            use_independent_randomness: true
                        - or_filter:
                            filters:
                            - status_code_filter:
                                comparison:
                                  op: LE
                                  value:
                                    default_value: 199
                                    runtime_key: http_err_response_sampling_status_le
                            - status_code_filter:
                                comparison:
                                  op: GE
                                  value:
                                    default_value: 201
                                    runtime_key: http_err_response_sampling_status_ge

Affected product area (please put an X in all that apply)
[ ] Ambient
[ ] Docs
[ ] Dual Stack
[ ] Installation
[ x] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Affected features (please put an X in all that apply)
[ ] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane
Additional context

This is something to discuss in Envoy repo, given:

Draining behavior is driven by Envoy
You are using raw EnvoyFilter
Note they will want the issues described purely in terms of Envoy though without Istio abstraction typically.


Recursive xml_tree

xml_tree should be rewritten to be recursive and thus accomodate an arbitrary number of levels

Issue solved. Closing now.


Add License

I'll Add license

Sure @Thrillseekr, next time make sure to follow the template when creating issue


SecurityTokenInvalidSignatureException encountered when using "AddDeveloperSigningCredential"

Question / Steps to reproduce the problem
I created an IdentitySever4 ASP.Net Core 3.1 website using the EF template (http://docs.identityserver.io/en/latest/quickstarts/5_entityframework.html) and modifying it (template created using .Net Core 3, but I upgrade it to 3.1 and upgraded all the NuGet packages as well).  It seeds a API Resource and Client with a secret for client-credentials acccess on startup.
I the identity configuration is signed using the AddDeveloperSigningCredential extension method (this is currently only local development).
I create an API project (the one linked to the API Resource that is seeded at startup) and configured it as mentioned in the quick start guide (http://docs.identityserver.io/en/latest/quickstarts/1_client_credentials.html).
I request the token from the Identity Server using client credentials and then try to do a post to the API, but it encounters a SecurityTokenInvalidSignatureException.
Working example
The full code is available at the tag: https://github.com/TheMagnificent11/identity-server-admin/tree/Github-SecurityTokenInvalidSignatureException-Issue
Relevant parts of the log file
info: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[1]
      Failed to validate the token.
Microsoft.IdentityModel.Tokens.SecurityTokenInvalidSignatureException: IDX10503: Signature validation failed. Keys tried: '[PII is hidden. For more details, see https://aka.ms/IdentityModel/PII.]'.
Exceptions caught:
 '[PII is hidden. For more details, see https://aka.ms/IdentityModel/PII.]'.
token: '[PII is hidden. For more details, see https://aka.ms/IdentityModel/PII.]'.
   at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.ValidateSignature(String token, TokenValidationParameters validationParameters)
   at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.ValidateToken(String token, TokenValidationParameters validationParameters, SecurityToken& validatedToken)
   at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()

did you follow https://aka.ms/IdentityModel/PII ?

I turned it on and it showed up some strange claims in the token.
Turns out the issue was my Postman collection.  I was saving the token in an environment variable and the same variable was set as global variable in another collection.
Thanks.


Update message keys for configuration.razor.trace

PR for #6893

@dotnet-policy-service agree

@Cosifne Was I wrong to think these should have automatically been updated by the localization process, as would happen with .xlf files in VS land? Is there some other part of the process I missed to clean up localization?

@davidwengier  the key here is auto-generated by vscode-l10n, not sure why it's out-off sync originally, let me check

if possible could you check if the config is still grouped with other razor configs?

I think this is what you are asking. There are other Razor settings in the LSP Server group, but they are not all together as one.

Razor > Server is not grouped with Razor > Language Server

When editing the settings JSON it also fails to translate the descriptions of the settings:

Out of curiosity, is there a difference between Razor > Server and Razor > Language Server?

Out of curiosity, is there a difference between Razor > Server and Razor > Language Server?

Naming is hard :)


Alpine/Musl with Pact Rust based core, in Pact client libraries

Background
Many teams choose to run builds on Alpine Linux due to its tiny footprint, and smaller security surface area.
Guidance for Alpine users, has been stored on our docs site here
For Pact implementations that use the Ruby shared core
Support is available by installing glibc compatibility layers due the underlying runtime being built on centos7
https://github.com/pact-foundation/pact-ruby-standalone/wiki/Using-the-pact-ruby-standalone-with-Alpine-Linux-Docker
For Pact implementations that use the Rust shared core
Alpine or other musl based distros have not be supported due to

Lack of musl based artifacts from pact-rust based projects
Potential issues requiring the project to possibly be rebuilt against specific versions of Alpine / musl.

Point 1 is now addressed, partially for the FFI library.
Pact-Reference is now publishing musl based versions of the Pact-FFI as of 0.4.15
We would like to investigate creating musl variants of all the pact-reference (rust) based artifacts, and incorporating them into client libraries
Support Level
Any of the musl based artifacts should be considered somewhat experimental and you may encounter issues, as they have not been used for a long time. I would strongly advise these are not used in production workflows, unless you are handy and don't mind investigating issues yourself
We would gratefully ask that if issues are raised, that two things are provided

A minimal reproducible example of a Docker image/file based on Alpine that recreates the issue
A minimal reproducible example of a Docker image/file based on Debian that recreates the issue

Note - The debian image, may not show the same behaviour, this will be useful to determine where there are bugs/issues or gaps in the musl implementation which can be communicated to others.
How to participate?
Drop a comment in this thread, or reach out via slack in #libpact_ffi-users
FFI
Investigate the possibility of incorporating into client libraries
Pact-Js-Core

[ ] Pact-Js-Core

[x] build musl pact.node bindings - Delta
[ ] release prebuilds
[X] CI Test runs

[X] arm64 musl
[X] amd64 musl

Snags

pact ruby standalone arm64 fail under qemu in GHA which fails test suite
some inconsistent behaviour in node 16 (set /tmp cache) / 18 ( max sockets 1)
prebuilds take a good while, but these could be limited to only running when the ffi, or native bindings are updated, or failing that just run on pull requests. regular test runs could just grab the latest pre build

Pact-JS

[ ] Pact-JS

Pact-PHP

[ ] Pact-PHP

[x] Support for musl - delta
[X] CI Test runs

[X] arm64 musl - gha
[X] amd64 musl - gha

Snags

PHP can only load .so files, so requires converted .a musl library
AMD64 musl images require shared-mime-info installing for consistent content type matching, ARM64 doesn't. If installed, they flip behaviour.
Needs musl detection method in downloader/

Pact-Go

[ ] Pact-Go

[x] Support for musl - delta
[ ] CI Test runs

[ ] arm64 musl - partially working, some seg faults
[ ] amd64 musl - partially working, some seg faults

Snags

Some seg fauls, also occur on vanilla x86_64 build although not consistently reproducible.

Pact-Net

[ ] Pact-Net

[x] Support for musl - delta
[X] CI Test runs

[ ] arm64 musl - gha - failing
[X] amd64 musl - gha
[X] arm64 musl - cirrus
[X] amd64 musl - cirrus

Snags

.NET can only load .so files, so requires converted .a musl library
ARM64 musl on GHA fails to build project, but passes on Cirrus
Needs musl detection method in csproj file

add others
Mock Server CLI

[ ] Create musl variants

[X] Fork release
[X] PR
[ ] Publish

Provider Verifier CLI

[ ] Create musl variants

[X] fork release
[X] PR
[ ] Publish

Stub Server CLI

[ ] Create musl variants

[X] fork release
[X] PR
[ ] Publish

Plugins
pact plugin cli

[x] fork release
[ ] issue building existing x86_64 linux executable in CI (build with cross to retain known glibc version)
[ ] pact plugin driver rust
[ ] pact plugin driver jvm
[ ] support downloading non musl variants if not avail, via pact-plugin-cli (rust and jvm driver)

Pact-Protobuf-Plugin

[ ] Create musl variants

[x] fork release
[ ] PR

Pact-csv-Plugin

[ ] Create musl variants
[x] fork release
[ ] PR

Pact-matt plugin
Build with goreleaser and linked with musl, so existing binaries work with alpine.

[x]  support downloading non musl variants if not avail, via pact-plugin-cli (rust and jvm driver)

Client language FFI bindings that only support dynamic *.so libaries
For some languages, it is not possible to load a *.a static library.

PHP
Ruby
Others?

It is possible however to convert a .a file to a .so file.
ar -x libpact_ffi.a
gcc -shared *.o -o libpact_ffi.so
rm -f *.o

This shrinks our .a file from 115mb to a circa 41mb .so file, which looks inline with the size of the musl node prebuild for pact-js

[X] Create .so musl variants of pact ffi

[X] Fork release
[X] Delta
[X] PR
[ ] Publish

So that was a successful endeavour however, we are going to try a different approach.
The musl built artifacts with work on systems based on glibc, therefore we technically only need to build a single linux binary linked to musl.
This will reduce the need for determining glibc/musl for at least rust built executables.
It is still to be tested with the FFI, to determine if we can get away with just one set (musl based for all)

Only two left to go are pact-net and pact-go, with RFC's/PR's raised.
Will close this off


Clean up redundant same-package ..package imports

Description:
SSIA, probably leftovers from tree restructuring.
Checklist:

[x] The code change is tested and works locally (huawei_lte only).
[ ] Local tests pass with tox. Your PR cannot be merged unless tests pass
[x] There is no commented out code in this PR.

If user exposed functionality or configuration variables are added/changed:

[ ] Documentation added/updated in home-assistant.io

If the code communicates with devices, web services, or third-party tools:

[ ] The manifest file has all fields filled out correctly (example).
[ ] New dependencies have been added to requirements in the manifest (example).
[ ] New or updated dependencies have been added to requirements_all.txt by running script/gen_requirements_all.py.
[ ] New files were added to .coveragerc.

If the code does not interact with devices:

[ ] Tests have been added to verify that the new code works.

Was circleci cancelled on purpose?

I stopped using circleci for my own builds, I had no idea that canceling a build there could possibly result in the cancellation being visible here as well (it says it's the scop->home-assistant project, not home-assistant->home-assistant). Well assuming that's what happened because circleci says I cancelled it.
I'll see if I can somehow restart the build (probably need to re-enable circleci for myself then ;(, at least temporarily).

Ok, it seems to be in progress now, let's see how it goes. BTW, see #23584 for some more circleci oddity.

CircleCI has marked test 3.6 as failed, but AFAICT everything except the end result there is green/success, 0 failures. Oh well.


Multer file validation

related to https://github.com/eddienubes/validness/issues/4

#22 feature is implemented


Bugfix for Timestamp Overflow in BigQueryIO's StorageWriteAPI

Current issue:
As per the specifications, BigQuery is capable of receiving timestamps up to 9999-12-31T23:59:59.
https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#timestamp_type
There is a bug in the java.time.Instant.until. https://bugs.java.com/bugdatabase/view_bug?bug_id=8199095
This causes there is an overflow issue with timestamps beyond 2262-04-11T23:47:16Z. This prevents correct processing of timestamps beyond this point.
https://github.com/apache/beam/blob/v2.48.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java#L764
ErrorMessage: org.apache.beam.sdk.io.gcp.bigquery.TableRowToStorageApiProto$SchemaDoesntMatchException: Problem converting field root.COLUMN_NAME expected type: TIMESTAMP. Exception: java.lang.ArithmeticException: long overflow

Details
Instant.until calculates the period using ChronoUnit.UNIT.between. ChronoUnit.MICROS.between will multiply 1000_000_000(=Instant.NANOS_PER_SECOND) to the micro-second based Instant object and convert it to nano-second based one.
https://github.com/frohoff/jdk8u-jdk/blob/master/src/share/classes/java/time/Instant.java#L1149
This means, the maximum timestamp which ChronoUnit.MICROS.between can handle is

long.max =<PHONE_NUMBER>854775807
Timestamp(long.max, UNIT.NANO) ~ 2262-04-11T23:47:16Z

TableRowToStorageApiProto.singularFieldToProtoValue will use ChronoUnit.MICROS.between(Instant.EPOCH, Instant.from(timestamp). This will be overflowed if timestamp > 2262-04-11T23:47:16Z and it throw java.lang.ArithmeticException: long overflow.

Proposed Fix:
This pull request applies a workaround to the section of code where the overflow occurs. With this fix, the StorageWriteAPI can correctly transmit timestamps up to 9999-12-31 23:59:59.999999 UTC as originally intended by BigQuery's specifications.
Workaround:
BigQueryIO can accept an Epoch timestamp in microsecond as long type. For example, "1970-01-01T00:01:01.000041Z" as string -> 61000041L.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

assign set of reviewers

It looks 2 workflows failed due to another reason.🤔

PreCommit Java Examples Dataflow -> GCP API error?
Java_GCP_IO_Direct -> Spanner ChangeStream got error

Does anyone know how to fix it? or xan I ignore them?

R: @ahmedabu98

FYI failing tests look irrelevant, you can ignore them


Weather App

Hi Sir/Madam ✋ ;
In this repo:

[ ] Used React components.
[ ]  Used React props.
[ ]  Used React Router.
[ ]  Connected React and Redux.
[ ]  Handle events in a React app.
[ ]  Write integration tests with a React testing library.
[ ]  Added styling for the home & details
[ ]  Applied React best practices and language style guides in code.
[ ]  Used store, actions, and reducers in React.
[ ]  Fix the Linter's errors
[ ]  Updated readme with professional documentation
[ ]  Checked that the app is working properly

Thank you ...Noted..I will fix the error

Hi 👋    Thank you for your feedback, I have implemented the following:

[ ] I have fixed the error on App.test.js.
[ ] I have made sure that all the test pass

As is shown below 😃

Thank you. 👍  😄


Update index.html

Changed ghost name "Mimic" to "The Mimic".
Spelling correction on line 806.
Added information on ghost types Deogen, Moroi, and Thaye.
(Took me a few tries to figure this committing thing out.  First time contributor on GH.)

Thanks State!

Thanks State!

Sure thing!  Thanks for developing this!  It's my favorite of all the ones I've seen.


chore: add multiple db platforms in GA workflow PHPUnit

Probably now Shield works with only MySQL or SQLite3.

[x] MySQL #336
[x] Postgre #339
[x] SQLSRV
[ ] OCI8

I was wrong. Tests failed on MySQL.
There was 1 error:

1) Tests\Authentication\Authenticators\SessionAuthenticatorTest::testForgetAnotherUser
mysqli_sql_exception: Incorrect datetime value: 'nesciunt' for column 'expires' at row 1

/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/Database/MySQLi/Connection.php:292
/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/Database/BaseConnection.php:695
/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/Database/BaseConnection.php:609
/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/Database/BaseBuilder.php:1904
/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/Model.php:278
/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/BaseModel.php:738
/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/Model.php:666
/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/Test/Fabricator.php:469
/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/Helpers/test_helper.php:38
/home/runner/work/shield/shield/tests/Authentication/Authenticators/SessionAuthenticatorTest.php:237
phpvfscomposer:///home/runner/work/shield/shield/vendor/phpunit/phpunit/phpunit:97

--

There were 2 failures:

1) Tests\Controllers\LoginTest::testLoginActionUsernameSuccess
The value of 'user' does not match expected value.
Failed asserting that two arrays are identical.
--- Expected
+++ Actual
@@ @@
 Array &0 (
-    'id' => 1
+    'id' => '1'
 )

/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/Test/TestResponse.php:264
/home/runner/work/shield/shield/tests/Controllers/LoginTest.php:139
phpvfscomposer:///home/runner/work/shield/shield/vendor/phpunit/phpunit/phpunit:97

2) Tests\Controllers\LoginTest::testLoginActionEmailSuccess
The value of 'user' does not match expected value.
Failed asserting that two arrays are identical.
--- Expected
+++ Actual
@@ @@
 Array &0 (
-    'id' => 1
+    'id' => '1'
 )

/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/Test/TestResponse.php:264
/home/runner/work/shield/shield/tests/Controllers/LoginTest.php:79
phpvfscomposer:///home/runner/work/shield/shield/vendor/phpunit/phpunit/phpunit:97

https://github.com/codeigniter4/shield/runs/7625886471?check_suite_focus=true

Run vendor/bin/phpunit --verbose --coverage-text --testsuite main
PHPUnit 9.5.21 #StandWithUkraine

Runtime:       PHP 7.4.30 with Xdebug 3.1.2
Configuration: /home/runner/work/shield/shield/phpunit.xml.dist
Random Seed:<PHONE_NUMBER>

...............................................................  63 / 183 ( 34%)
............................................................... 126 / 183 ( 68%)
.........................................................       183 / 183 (100%)

Nexus\PHPUnit\Extension\Tachycardia identified these 3 slow tests:
+--------------------------------------------------------------------------+---------------+-------------+
| Test Case                                                                | Time Consumed | Time Limit  |
+--------------------------------------------------------------------------+---------------+-------------+
| Tests\\Controllers\\RegisterTest::testRegisterRedirectsToActionIfDefined | 00:00:01.01   | 00:00:00.50 |
| Tests\\Controllers\\RegisterTest::testRegisterActionSuccess              | 00:00:01.01   | 00:00:00.50 |
| Tests\\Unit\\DictionaryValidatorTest::testCheckTrueOnNotFound            | 00:00:00.74   | 00:00:00.50 |
+--------------------------------------------------------------------------+---------------+-------------+

Warning: Took 1.01s from 0.50s limit to run Tests\\Controllers\\RegisterTest::testRegisterRedirectsToActionIfDefined
Warning: Took 1.01s from 0.50s limit to run Tests\\Controllers\\RegisterTest::testRegisterActionSuccess
Warning: Took 0.74s from 0.50s limit to run Tests\\Unit\\DictionaryValidatorTest::testCheckTrueOnNotFound

Time: 00:23.047, Memory: 78.00 MB

OK (183 tests, 471 assertions)

PHP 8.0 - OCI8
There were 26 errors:

1) Tests\Authentication\Authenticators\AccessTokenAuthenticatorTest::testLoginByIdNoToken
ErrorException: oci_execute(): ORA-01400: cannot insert NULL into ("ORACLE"."db_auth_token_logins"."identifier")

/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/Database/OCI8/Connection.php:198
/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/Database/BaseConnection.php:693
/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/Database/BaseConnection.php:620
/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/Database/BaseBuilder.php:1916
/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/CodeIgniter.php:340
/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/Test/FeatureTestTrait.php:189
/home/runner/work/shield/shield/tests/Authentication/Filters/ChainFilterTest.php:23

--

There was 1 failure:

1) Tests\Authentication\Authenticators\SessionAuthenticatorTest::testAttemptUserHavingSessionDataAttemptsAgain
Failed asserting that exception of type "ErrorException" matches expected exception "CodeIgniter\Shield\Exceptions\LogicException". Message was: "oci_execute(): ORA-00904: "DB_AUTH_IDENTITIES"."SECRET": invalid identifier" at
/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/Database/OCI8/Connection.php:198
/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/Database/BaseConnection.php:693
/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/Database/BaseConnection.php:620
/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/Database/BaseBuilder.php:1585
/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/Model.php:249
/home/runner/work/shield/shield/vendor/codeigniter4/framework/system/BaseModel.php:622
/home/runner/work/shield/shield/src/Models/UserModel.php:183
/home/runner/work/shield/shield/src/Authentication/Authenticators/Session.php:287
/home/runner/work/shield/shield/src/Authentication/Authenticators/Session.php:103
/home/runner/work/shield/shield/tests/Authentication/Authenticators/SessionAuthenticatorTest.php:369
.

ERRORS!
Tests: 183, Assertions: 372, Errors: 26, Failures: 1.

All green,
Do we need to list the databases that Shield supports in the documentation?

Do we need to list the databases that Shield supports in the documentation?

It is better to have it.
If you send a PR, please mention MySQL (InnoDB).


Query about licensing

Hi WesleyE,
This is fun, I like and appreciate what you've put together here.
Which license have you decided on for the hardware design?
Same question but for your code?
Thanks!

Hi! I just added https://github.com/WesleyE/AirQuality_Monitor/blob/main/LICENSE. Honestly, I would not use it for anything else than personal use. If I were to have time to build a v4, there are quite a few things I would do differently.


How to use it in other cases?

I would like to use it in my site, and I wonder if it can let me input the file path by myself?
e.g.: start/contributor.md

oh the more serious problem is that when I use it in an org repo, it failed like this:

i suppose this may indicates that the github token is not valid
any solutions will help

Hey @CLCK0622 .
Yes u can change the file path. All you do is set the option readme_path while setting the action.
Ref: https://github.com/akhilmhdh/contributors-readme-action#optional-parameters

@CLCK0622 can u share me the repo url so that I can take a look at it.
Now sure, have seen users configuring it for organization and was working great for them

hey i am having same problem, i have already given readme_path here's repo can you take a look? https://github.com/Prakash4844/Get-Pair-Extraordinaire

Hey @Prakash4844 Thank you for taking your time. I really appreciate it.

Weird i must say because the action uses node16. Ref: https://github.com/akhilmhdh/contributors-readme-action/blob/master/action.yml#L46 Need to check why its throwing an error like that.

Ill definitely check you repo and resolve this. Release a patch. I got busy recently with some career changes happening.

Ill raise the witch within a couple of days. v3 will be released next month which would be an amazing one.

Thanks, eagerly waiting for new version. keep up the good work

I also had this issue today
https://github.com/iChristwin/M3ter-head/actions/runs/7340215377/job/19986281011

@Prakash4844 @iChristwin This is because you Github changed there action permission, You need to allow github action to read and write to repo.
I have updated the readme usage workflow to enable this.
Apologies for the very delayed response.


fix: allow expired version timestamps to return as None

Not sure the None return from this function is ideal. Other functions might expect to get a timestamp in return


Swagger fixes (task #5711)

Added Swagger documentation for delete end-point
Extended the Swagger logic to exclude modules without migration config
Dynamically generate Swagger Info annotation

Codecov Report

Merging #535 into master will decrease coverage by 0.2%.
The diff coverage is 0%.

@@             Coverage Diff              @@
##             master     #535      +/-   ##
============================================
- Coverage     28.13%   27.93%   -0.21%
- Complexity      954      960       +6
============================================
  Files            85       85
  Lines          3252     3276      +24
============================================
  Hits            915      915
- Misses         2337     2361      +24

Impacted Files
Coverage Δ
Complexity Δ

src/Controller/Api/V1/V0/AppController.php
46.26% <0%> (-2.43%)
38 <6> (+4)

src/Swagger/Annotation.php
0% <0%> (ø)
17 <3> (+2)
:arrow_up:

src/Swagger/Analyser.php
0% <0%> (ø)
2 <0> (ø)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 63f2fb2...ae26016. Read the comment docs.


add index page

This PR adds an index page with information about the generated metrics

closing in favor of more changes coming soon


Improve settings dialog

What does this implement/fix?
This improves the settings dialog by:

[x] Adding an antialiasing checkbox
[ ] adding a toggle for theme (as mentioned in #83)
[ ] <other suggestions of parameters to expose?>

@marsipu should we just merge this as-is and proceed with follow-up enhancements for the other checkboxes later?

@larsoner yes, sorry forgot about this one.
Last thing I tried was switching while the app was running but I didn't succeed.
That could be subject of another PR together with adding the setting.
Could you unlink the dark-theme-issue before you merge so it doesn't get closed?
Thx

@larsoner yes, sorry forgot about this one.
Last thing I tried was switching while the app was running but I didn't succeed.
That could be subject of another PR together with adding the setting.
I am currently away from my PC but I can look at the merge conflicts next week

Marking for merge when green, thanks in advance @marsipu !


FIXED: Functions that had a destructured argument as well as a type checked return would fail transpilation.

fixes issue #16

Amazing, another long standing issue solved ;-)


DisplayPromptAsync not implemented for GTK

Xamarin.Forms.Platform.GTK (4.5 & 4.6) doesn't implement the DisplayPromptAsync feature

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 3c6e0bb1-26e4-091b-8a99-27d772bed7f6
Version Independent ID: 57d8ad92-0aa2-9ec4-47bc-8853d9a71d91
Content: Display Pop-ups - Xamarin
Content Source: docs/xamarin-forms/user-interface/pop-ups.md
Product: xamarin
Technology: xamarin-forms
GitHub Login: @davidbritch
Microsoft Alias: dabritch

Hi @Grayhead1
Afraid that's the case. The GTK# support in Xamarin.Forms is still classed as a preview, and is a community ran project. New functionality is only added as and when community members have time.


Custom Network

Hello, I am trying to implement a custom network and want to use a json file to configure the agent. Say it is XNet whose super class is LayerBasedNetwork which I append to tensorforce.core.networks.layers. In order to call this, I add
{"type": XNet, layers=[...]} to the json and feed this to Agent.from_spec as network. However, it does not call my XNet implementation. How can I resolve this?

Hi @hfurkanbozkurt, could you please paste the code where you call Agent.from_spec(...) including the specification? Or maybe this helps already: If you specify a network, there is no need to add it to layers, but instead just pass your dict {"type": X, ...} to the argument network={...} directly.

Hi @AlexKuhnle, thank you so much for your prompt answer. I have resolved the problem. Instead of giving just "type": "XNet", I should have given "type": "tensorforce_extras.XNet". Then it works like that. As an example, I am adding the code:
import json

from tensorforce.agents import Agent
from tensorforce.contrib.openai_gym import OpenAIGym

from tensorforce.core.networks import LayeredNetwork

class XNet(LayeredNetwork):
    def __init__(self, *args, **kwargs):
        print("This is XNet")
        super(XNet, self).__init__(*args, **kwargs)

with open('custom.json') as tensorforce_spec:
    spec = json.load(tensorforce_spec)

environment = OpenAIGym(
    gym_id=spec["environment"]
)

agent = Agent.from_spec(
    spec=spec["agent"],
    kwargs=dict(
        states=environment.states,
        actions=environment.actions,
        network=spec["network"],
    )
)

with config:
{
    "agent": {
        "type": "vpg_agent",

        "update_mode": {
            "unit": "episodes",
            "batch_size": 20,
            "frequency": 20
        },
        "memory": {
            "type": "latest",
            "include_next_states": false,
            "capacity": 5000
        },

        "optimizer": {
            "type": "adam",
            "learning_rate": 2e-2
        },

        "discount": 0.99,
        "entropy_regularization": null,
        "gae_lambda": null,

        "baseline_mode": "states",
        "baseline": {
            "type": "mlp",
            "sizes": [32, 32]
        },
        "baseline_optimizer": {
            "type": "multi_step",
            "optimizer": {
                "type": "adam",
                "learning_rate": 1e-3
            },
            "num_steps": 5
        },

        "saver": {
            "directory": null,
            "seconds": 600
        },
        "summarizer": {
            "directory": null,
            "labels": ["graph", "total-loss"]
        },
        "execution": {
            "type": "single",
            "session_config": null,
            "distributed_spec": null
        }
    },
    "environment": "CartPole-v0",
    "network": {
        "type": "test_tensorforce.XNet",
        "layers": [
            {
                "type": "dense",
                "size": 100,
                "activation": "relu"
            },
            {
                "type": "dense",
                "size": 50,
                "activation": "relu"
            }
        ]
    }
}


$stateChangeSuccess and $ionicHistory wrong $stateParams

I came across an issue that puzzeld me for 2 days now.
It consists of 2 issues but they may be related to the same rootcause.
I configured 2 states like this:
	.state('tab', {
		url: "/tab",
		abstract: true,
		templateUrl: "templates/menu.html",
		controller: "MenuCtrl"
	})
	.state('tab.location', {
		cache: false,
		url: '/beacon',
		params: {beaconUser:{}, beacon:{}},
		views: {
			'menuContent': {
				templateUrl: 'templates/Beacon.html',
				controller: 'BeaconCtrl'
			}
		},
	})
       .state('tab.chat', {
	     cache: false,
    	     url: '/chat',
	     params: {chatUserId:{}},
    	     views: {
    		    'menuContent': {
    			    templateUrl: 'templates/Chat.html',
    			    controller: 'ChatCtrl'
    		    }
    	     }
        })

First I navigate to the location state like this:
ui-sref="tab.location({locationUser: user})"
```
and from there to the chat state like this:

ui-sref="tab.chat({chatUserId: userid})"

First issue:
registering a listener to $stateChangeSuccess during the run method of a module

$rootScope.$on('$stateChangeSuccess',function(event, toState, toParams, fromState, fromParams){
	if(toState.name == "tab.chat"){
		$ionicHistory.backView().stateParams //undefined
	};
});

note that $ionicHistory.backView().stateParams is undefined. Whereas registering the same listener in a controller

$scope.$on('$stateChangeSuccess',function(event, toState, toParams, fromState, fromParams){
	if(toState.name == "tab.chat"){
		$ionicHistory.backView().stateParams //defined
	};
});

$ionicHistory.backView().stateParams is existing!

On to the second issue...

Even on the controller level the stateParams in history are incorrect. I would expect them to be exactly the same as the fromParams, but they aren't.
this offcourse leeds to an incorrect behaviour of my app when pressing the back button and going back to the location state.
For the time being I fixed that by

$scope.$on('$stateChangeSuccess',function(event, toState, toParams, fromState, fromParams){
	if(toState.name == "tab.chat"){
		$ionicHistory.backView().stateParams = fromParams
	};
});

Greetings @akreienbring!
My sensors indicate that you need to update your issue through our custom issue form. We are now requiring all issues to be submitted this way, to ensure that we have all of the information necessary to fix them as quickly as possible.
Click Here To Update Your Issue
I will have no choice but to close this issue if it is not resubmitted through the form. Please fill out the rest of the form, so that I may use my friendly robot powers to assist you.
Thank you!

Hi,
I just created a new issue that may be related that involves incorrect $stateParams when using cached views that shows up in the $stateChangeStart and $ionicView.enter events:
https://github.com/driftyco/ionic/issues/3884
Cheers,
Robert

Just checked the behaviour with version 1.2.1. Still the same :-(

Hello all! Is this still an issue with the latest version of ionic 1? (1.3.0)

Hello all! As it seems it has been a while since there was any activity on this issue i will be closing it for now. Feel free to comment if you are still running into this issue. Thanks for using Ionic!


Change health workspace service title

Resolves #3110 3110
What is being addressed
Change health workspace service title

/test-force-approve


Detail using specific components in an MDXProvider

In the docs:
import React from 'react';
import { MDXEmbedProvider } from 'mdx-embed';

export default (props) => <MDXEmbedProvider>{props.children}</MDXEmbedProvider>;

Say if the user only wants to use the the <CodePen /> and <Gist /> components, they're not tied into using the MDXEmbedProvider so I think it should be made clear, something along the lines of:
Only want to use a couple of components from MDX Embed, you can import only the components you need into the MDXProvider
import React from 'react';
import { MDXProvider } from '@mdx-js/react'
import { CodePen, Gist } from 'mdx-embed';

const components = {
  CodePen,
  Gist
}

export default (props) => <MDXProvider components={components}>{props.children}</MDXProvider>;

@spences10 yeah lovely idea mate!

addressed in [#90]


PlatformIO breaks serial interface for Arduino Leonardo (but only for PlatformIO)

When I flash my Leonardo with your software, it will start disconnecting from serial immediately after having been connected. It will only very briefly appear as a tty serial device when being connected, and not long enough to flash it using PlatformIO.
Luckily the Arduino IDE can still flash it (I have open the "Tools" menu in the 300ms after I've reset the Arduino, select the right port, then press the reset button again while uploading). After the Arduino IDE has flashed it it stays connected via serial again, and PlatformIO can flash it again.
Here's some outputs:
ford@FordPrefect:~/Documents/work/42tech/cbgb/arduino$ platformio run --upload-port "/dev/ttyACM0"
[Thu Apr 16 02:25:48 2015] Processing leonardo (targets: upload, platform: atmelavr, board: leonardo, framework: Arduino)
-----------------------------------------------------------------------------------
BeforeUpload(["upload"], [".pioenvs/leonardo/firmware.hex"])
scons: *** [upload] /dev/ttyACM0: No such file or directory
=========================== [ ERROR ] Took 2.39 seconds ===========================
ford@FordPrefect:~/Documents/work/42tech/cbgb/arduino$ platformio run --upload-port "/dev/ttyACM0"
[Thu Apr 16 02:25:53 2015] Processing leonardo (targets: upload, platform: atmelavr, board: leonardo, framework: Arduino)
-----------------------------------------------------------------------------------
BeforeUpload(["upload"], [".pioenvs/leonardo/firmware.hex"])
scons: *** [upload] Broken pipe
=========================== [ ERROR ] Took 2.40 seconds ===========================

When I have just flashed the Leonardo with the Arduino IDE, I still have to press the reset button when it's started uploading. Here's the output that goes along with that:
ford@FordPrefect:~/Documents/work/42tech/cbgb/arduino$ platformio run --upload-port "/dev/ttyACM0"
[Thu Apr 16 02:25:31 2015] Processing leonardo (targets: upload, platform: atmelavr, board: leonardo, framework: Arduino)
-----------------------------------------------------------------------------------
BeforeUpload(["upload"], [".pioenvs/leonardo/firmware.hex"])
"/home/ford/.platformio/packages/tool-avrdude/avrdude" -v -D -p atmega32u4 -C "/home/ford/.platformio/packages/tool-avrdude/avrdude.conf" -c avr109 -b 57600 -P /dev/ttyACM0 -U flash:w:.pioenvs/leonardo/firmware.hex:i

avrdude: Version 6.0.1, compiled on Apr  3 2014 at 21:52:43
Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
Copyright (c) 2007-2009 Joerg Wunsch

System wide configuration file is "/home/ford/.platformio/packages/tool-avrdude/avrdude.conf"

Using Port                    : /dev/ttyACM0
Using Programmer              : avr109
Overriding Baud Rate          : 57600
AVR Part                      : ATmega32U4
Chip Erase delay              : 9000 us
PAGEL                         : PD7
BS2                           : PA0
RESET disposition             : dedicated
RETRY pulse                   : SCK
serial program mode           : yes
parallel program mode         : yes
Timeout                       : 200
StabDelay                     : 100
CmdexeDelay                   : 25
SyncLoops                     : 32
ByteDelay                     : 0
PollIndex                     : 3
PollValue                     : 0x53
Memory Detail                 :

Block Poll               Page                       Polled
Memory Type Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
eeprom        65    20     4    0 no       1024    4      0  9000  9000 0x00 0x00
flash         65     6   128    0 yes     32768  128    256  4500  4500 0x00 0x00
lfuse          0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
hfuse          0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
efuse          0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
lock           0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
calibration    0     0     0    0 no          1    0      0     0     0 0x00 0x00
signature      0     0     0    0 no          3    0      0     0     0 0x00 0x00

Programmer Type : butterfly
Description     : Atmel AppNote AVR109 Boot Loader

Connecting to programmer: .
Found programmer: Id = "CATERIN"; type = S
Software Version = 1.0; No Hardware Version given.
Programmer supports auto addr increment.
Programmer supports buffered memory access with buffersize=128 bytes.

Programmer supports the following devices:
Device code: 0x44

avrdude: devcode selected: 0x44
avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude: Device signature = 0x1e9587
avrdude: safemode: lfuse reads as FF
avrdude: safemode: hfuse reads as D8
avrdude: safemode: efuse reads as CB
avrdude: reading input file ".pioenvs/leonardo/firmware.hex"
avrdude: writing flash (240 bytes):

Writing | ################################################## | 100% 0.02s

avrdude: 240 bytes of flash written
avrdude: verifying flash memory against .pioenvs/leonardo/firmware.hex:
avrdude: load data flash data from input file .pioenvs/leonardo/firmware.hex:
avrdude: input file .pioenvs/leonardo/firmware.hex contains 240 bytes
avrdude: reading on-chip flash data:

Reading | ################################################## | 100% 0.00s

avrdude: verifying ...
avrdude: 240 bytes of flash verified

avrdude: safemode: lfuse reads as FF
avrdude: safemode: hfuse reads as D8
avrdude: safemode: efuse reads as CB
avrdude: safemode: Fuses OK (H:CB, E:D8, L:FF)

avrdude done.  Thank you.

=========================== [SUCCESS] Took 5.53 seconds ===========================

I hope you can do something about this. To me this seems like you use a bugged/outdated firmware version. Feel free to contact me.

Could ask you to try upload firmware without --upload-port? Like:
platformio run --target upload

PlatformIO will automatically detect upload port.

I have. Here's the command run twice without --upload-port:
ford@FordPrefect:~/Documents/work/42tech/cbgb/arduino$ platformio run
[Wed Apr 22 21:37:39 2015] Processing leonardo (targets: upload, platform: atmelavr, board: leonardo, framework: Arduino)
--------------------------------------------------------------------------------
BeforeUpload(["upload"], [".pioenvs/leonardo/firmware.hex"])
Auto-detected UPLOAD_PORT: /dev/ttyACM0
"/home/ford/.platformio/packages/tool-avrdude/avrdude" -v -D -p atmega32u4 -C "/home/ford/.platformio/packages/tool-avrdude/avrdude.conf" -c avr109 -b 57600 -P /dev/ttyACM0 -U flash:w:.pioenvs/leonardo/firmware.hex:i

avrdude: Version 6.0.1, compiled on Apr  3 2014 at 21:52:43
Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
Copyright (c) 2007-2009 Joerg Wunsch

System wide configuration file is "/home/ford/.platformio/packages/tool-avrdude/avrdude.conf"

Using Port                    : /dev/ttyACM0
Using Programmer              : avr109
Overriding Baud Rate          : 57600
AVR Part                      : ATmega32U4
Chip Erase delay              : 9000 us
PAGEL                         : PD7
BS2                           : PA0
RESET disposition             : dedicated
RETRY pulse                   : SCK
serial program mode           : yes
parallel program mode         : yes
Timeout                       : 200
StabDelay                     : 100
CmdexeDelay                   : 25
SyncLoops                     : 32
ByteDelay                     : 0
PollIndex                     : 3
PollValue                     : 0x53
Memory Detail                 :

Block Poll               Page                       Polled
Memory Type Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
eeprom        65    20     4    0 no       1024    4      0  9000  9000 0x00 0x00
flash         65     6   128    0 yes     32768  128    256  4500  4500 0x00 0x00
lfuse          0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
hfuse          0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
efuse          0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
lock           0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
calibration    0     0     0    0 no          1    0      0     0     0 0x00 0x00
signature      0     0     0    0 no          3    0      0     0     0 0x00 0x00

Programmer Type : butterfly
Description     : Atmel AppNote AVR109 Boot Loader

Connecting to programmer: .
Found programmer: Id = "CATERIN"; type = S
Software Version = 1.0; No Hardware Version given.
Programmer supports auto addr increment.
Programmer supports buffered memory access with buffersize=128 bytes.

Programmer supports the following devices:
Device code: 0x44

avrdude: devcode selected: 0x44
avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude: Device signature = 0x1e9587
avrdude: safemode: lfuse reads as FF
avrdude: safemode: hfuse reads as D8
avrdude: safemode: efuse reads as CB
avrdude: reading input file ".pioenvs/leonardo/firmware.hex"
avrdude: writing flash (240 bytes):

Writing | ################################################## | 100% 0.02s

avrdude: 240 bytes of flash written
avrdude: verifying flash memory against .pioenvs/leonardo/firmware.hex:
avrdude: load data flash data from input file .pioenvs/leonardo/firmware.hex:
avrdude: input file .pioenvs/leonardo/firmware.hex contains 240 bytes
avrdude: reading on-chip flash data:

Reading | ################################################## | 100% 0.00s

avrdude: verifying ...
avrdude: 240 bytes of flash verified

avrdude: safemode: lfuse reads as FF
avrdude: safemode: hfuse reads as D8
avrdude: safemode: efuse reads as CB
avrdude: safemode: Fuses OK (H:CB, E:D8, L:FF)

avrdude done.  Thank you.

========================= [SUCCESS] Took 5.62 seconds =========================

That was the first time. As you can see it was succesful. However, when I run it again, it does this:
ford@FordPrefect:~/Documents/work/42tech/cbgb/arduino$ platformio run
[Wed Apr 22 21:37:56 2015] Processing leonardo (targets: upload, platform: atmelavr, board: leonardo, framework: Arduino)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BeforeUpload(["upload"], [".pioenvs/leonardo/firmware.hex"])
scons: *** [upload] Explicit exit, status Error: Please specify `upload_port` for environment or use global `--upload-port` option.
For the some development platforms it can be USB flash drive

Error: Please specify `upload_port` for environment or use global `--upload-port` option.
For the some development platforms it can be USB flash drive

===================================================================== [ ERROR ] Took 2.33 seconds =====================================================================

It means that problem occurs from the 2-nd attempt to upload firmware?

Correct. After the (Arduino) Leonardo has been written to by PlatformIO it not longer lists as a serial device at /dev/ for longer than half a second after connecting via USB. The Arduino IDE still manages to write to it in this short timeframe or somehow prevents the device from disconnecting from serial. PlatformIO doesn't detect the device or when timed right (plug in or reset the device a bit after BeforeUpload['upload'] appears) it says this:
scons: *** [upload] Broken pipe

On a side note: thanks for the quick response. PlatformIO still seems a lot more comfortable in use than the official Arduino tooling IMO, because it's more flexible (doesn't dictate the use of one specific project structure).

Thanks! I suppose where this issue maybe hidden. I've just created new branch-feature where we will try to resolve this issue. Please try this first build:
pip uninstall platformio
pip install https://github.com/platformio/platformio/archive/feature/issue-178.zip

No, sorry, this hasn't changed anything as far as I can see. To me it would seem that the problem is in the behavior of the Leonardo after having been flashed with PlatformIO, as it really does behave differently from the 'flashed by Arduino IDE"-way. Does PlatformIO upload firmware for the device aside from the code you give as an input? I'd suspect that to be the problem, but I'm pretty new to this.
 Check that you have the correct port selected. If it is correct, try pressing the board's reset button after initiating the upload.
Error: Couldn't find a board on the selected port. Check that you have the correct port selected. If it is correct, try pressing the board's reset button after initiating the upload.
========================= [ ERROR ] Took 14.46 seconds =========================

Run 2 - success!
ford@FordPrefect:~/Documents/work/42tech/cbgb/arduino$ platformio run
[Thu Apr 23 01:49:42 2015] Processing leonardo (targets: upload, platform: atmelavr, board: leonardo, framework: Arduino)
--------------------------------------------------------------------------------
BeforeUpload(["upload"], [".pioenvs/leonardo/firmware.hex"])
Auto-detected UPLOAD_PORT: /dev/ttyACM0
"/home/ford/.platformio/packages/tool-avrdude/avrdude" -v -D -p atmega32u4 -C "/home/ford/.platformio/packages/tool-avrdude/avrdude.conf" -c avr109 -b 57600 -P /dev/ttyACM0 -U flash:w:.pioenvs/leonardo/firmware.hex:i

avrdude: Version 6.0.1, compiled on Apr  3 2014 at 21:52:43
Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
Copyright (c) 2007-2009 Joerg Wunsch

System wide configuration file is "/home/ford/.platformio/packages/tool-avrdude/avrdude.conf"

Using Port                    : /dev/ttyACM0
Using Programmer              : avr109
Overriding Baud Rate          : 57600
AVR Part                      : ATmega32U4
Chip Erase delay              : 9000 us
PAGEL                         : PD7
BS2                           : PA0
RESET disposition             : dedicated
RETRY pulse                   : SCK
serial program mode           : yes
parallel program mode         : yes
Timeout                       : 200
StabDelay                     : 100
CmdexeDelay                   : 25
SyncLoops                     : 32
ByteDelay                     : 0
PollIndex                     : 3
PollValue                     : 0x53
Memory Detail                 :

Block Poll               Page                       Polled
Memory Type Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
eeprom        65    20     4    0 no       1024    4      0  9000  9000 0x00 0x00
flash         65     6   128    0 yes     32768  128    256  4500  4500 0x00 0x00
lfuse          0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
hfuse          0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
efuse          0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
lock           0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
calibration    0     0     0    0 no          1    0      0     0     0 0x00 0x00
signature      0     0     0    0 no          3    0      0     0     0 0x00 0x00

Programmer Type : butterfly
Description     : Atmel AppNote AVR109 Boot Loader

Connecting to programmer: .
Found programmer: Id = "CATERIN"; type = S
Software Version = 1.0; No Hardware Version given.
Programmer supports auto addr increment.
Programmer supports buffered memory access with buffersize=128 bytes.

Programmer supports the following devices:
Device code: 0x44

avrdude: devcode selected: 0x44
avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude: Device signature = 0x1e9587
avrdude: safemode: lfuse reads as FF
avrdude: safemode: hfuse reads as D8
avrdude: safemode: efuse reads as CB
avrdude: reading input file ".pioenvs/leonardo/firmware.hex"
avrdude: writing flash (240 bytes):

Writing | ################################################## | 100% 0.02s

avrdude: 240 bytes of flash written
avrdude: verifying flash memory against .pioenvs/leonardo/firmware.hex:
avrdude: load data flash data from input file .pioenvs/leonardo/firmware.hex:
avrdude: input file .pioenvs/leonardo/firmware.hex contains 240 bytes
avrdude: reading on-chip flash data:

Reading | ################################################## | 100% 0.00s

avrdude: verifying ...
avrdude: 240 bytes of flash verified

avrdude: safemode: lfuse reads as FF
avrdude: safemode: hfuse reads as D8
avrdude: safemode: efuse reads as CB
avrdude: safemode: Fuses OK (H:CB, E:D8, L:FF)

avrdude done.  Thank you.

========================= [SUCCESS] Took 4.94 seconds =========================

Run 3 - same problem :/
ford@FordPrefect:~/Documents/work/42tech/cbgb/arduino$ platformio run
[Thu Apr 23 01:49:50 2015] Processing leonardo (targets: upload, platform: atmelavr, board: leonardo, framework: Arduino)
--------------------------------------------------------------------------------
BeforeUpload(["upload"], [".pioenvs/leonardo/firmware.hex"])
scons: *** [upload] Explicit exit, status Error: Please specify `upload_port` for environment or use global `--upload-port` option.
For the some development platforms it can be USB flash drive

Error: Please specify `upload_port` for environment or use global `--upload-port` option.
For the some development platforms it can be USB flash drive

========================= [ ERROR ] Took 2.31 seconds =========================
ford@FordPrefect:~/Documents/work/42tech/cbgb/arduino$ platformio run

Let's rephrase what do we have:

PlatformIO can upload firmware to Leonardo board if it was previously flashed by Arduino IDE?
See Arduino IDE upload code / PlatformIO upload code and atmelavr platform.

Could you provide TeamViewer access to your Mac with connected Leonardo board? Because we don't have this board and we can't reproduce this issue.

Could I ask to run this command when you connect Leonardo to USB:
platformio serialports list

PlatformIO uses this command to auot-detect available upload ports.

That seems correct, yes. However, I do not have a Mac, I run Lubuntu 14.04. I'll have a look at the sources, and yes, I've just installed TeamViewer so give me a sign when you have the time.

P.S: @WillemWillemWillem please email to<EMAIL_ADDRESS>

The issue's solved, uploads work without resets now. There were two problems:

Not using Arduino.h in my project, which lists the Arduino Leonardo as a serial device. Without Arduino.h the Arduino Leonardo will still register as a serial device, but will disconnect less than a second after connecting/resetting.
A bug in PlatformIO making a reset of the Arduino Leonardo required during upload, fixed in this version:
https://github.com/platformio/platformio/archive/feature/issue-178.zip
Which can be installed using pip install [the URL above].
The changes in this version will be merged into PlatformIO 1.5.0 soon by @valeros.

Hi @WillemMali,
Can you please provide more details regarding Arduino.h. I'm trying to avoid including Arduino specific files or only want to include what's absolutely necessary. Can you explain what is happening and what code in Arduino.h (or files that are included indirectly) is preventing that from happening.
Also, I have two Leonardos and now broke both of them this way. Unfortunately, I just can't get anything running with the Reset button - with both Leonardos. I tried to use the Reset button to upload software without the 1200bps trick (that the Arduino IDE and PlatformIO do automatically) and I tried to use the button to get the microcontroller back into a programmable state as described above by you. I can't seem to ever see the device again with platformio serialports list or in Arduino IDE, even shortly after pressing the button or plugging the USB cable back in.
I use Linux Mint 17.
Cheers,
Andreas

Hi Andreas,
I personally am under the impression that Arduino.h puts the Arduino in a
permanent 'ready to flash software' state, which makes uploading a lot
smoother (no reset needed like you normally would). There's no easy way (I
have found) to extract the specific part of Arduino.h which does this, and
as it's a HUGE include it'll be hard to find the right part.
Could you maybe share (a stripped down version of) the code you've used to
brick your Arduino?  Then I'll have some more motivation to find a way
to fix this, and it'll be easier to play around with (if it is possible to
undo, which I'd expect).
As an aside, have you tried uploading the blink example using the Arduino
IDE, and fiddling with the timing of pressing the reset button? Getting
your Arduino's running again in some way will make debugging this a lot
easier. It worked for me, by only releasing the reset button something like
3 seconds after pressing the upload button. Try to time it with the outputs
you get from the console and fiddle.
That's all I have for now, good luck!
Willem
2015-12-01 13:05 GMT+01:00 andreas271828<EMAIL_ADDRESS>
Hi @WillemMali https://github.com/WillemMali,
Can you please provide more details regarding Arduino.h. I'm trying to
avoid including Arduino specific files or only want to include what's
absolutely necessary. Can you explain what is happening and what code in
Arduino.h (or files that are included indirectly) is preventing that from
happening.
Also, I have two Leonardos and now broke both of them this way.
Unfortunately, I just can't get anything running with the Reset button -
with both Leonardos. I tried to use the Reset button to upload software
without the 1200bps trick (that the Arduino IDE and PlatformIO do
automatically) and I tried to use the button to get the microcontroller
back into a programmable state as described above by you. I can't seem to
ever see the device again with platformio serialports list or in Arduino
IDE, even shortly after pressing the button or plugging the USB cable back
in.
I use Linux Mint 17.
Cheers,
Andreas
—
Reply to this email directly or view it on GitHub
https://github.com/platformio/platformio/issues/178#issuecomment-160947805
.

Thanks Willem!
It's a huge challenge to press and release the Reset button at the right time and then also click the Upload button at the right time or even select a different port within the short time given, because sometimes the port just keeps changing! But .... I got back access to both Leonardos :)
You mentioned one of the main reasons why I actually want to get rid of Arduino.h and the rest of the framework: it's HUGE ;) The second reason is that it's less portable.
My code's working fine without it (just using core AVR includes), but it's breaking the serial interface every time, even when I include Arduino.h. If you use sketches it executes some extra code that's not executed otherwise (see main.cpp in the relevant framework code). I'll see if I can clean that up and use it directly or if I have to go with the extra load for now.
Thanks again,
Andreas

Hi Andreas,
No problem at all! I'm glad you mannaged to unbrick them :)
If you do find out what piece of code in Arduino.h does this I'd be very interested; I personally couldn't find it and went with Arduino.h (and the longer compile times), but it'd be a relief to be rid of it.
Regards,Willem-------- Original message --------From: andreas271828<EMAIL_ADDRESS>Date: 12/02/2015  08:11  (GMT+01:00) To: platformio/platformio<EMAIL_ADDRESS>Cc: WillemMali<EMAIL_ADDRESS>Subject: Re: [platformio] PlatformIO breaks serial interface for Arduino
Leonardo (but only for PlatformIO) (#178)
Thanks Willem!
It's a huge challenge to press and release the Reset button at the right time and then also click the Upload button at the right time or even select a different port within the short time given, because sometimes the port just keeps changing! But .... I got back access to both Leonardos :)
You mentioned one of the main reasons why I actually want to get rid of Arduino.h and the rest of the framework: it's HUGE ;) The second reason is that it's less portable.
My code's working fine without it (just using core AVR includes), but it's breaking the serial interface every time, even when I include Arduino.h. If you use sketches it executes some extra code that's not executed otherwise (see main.cpp in the relevant framework code). I'll see if I can clean that up and use it directly or if I have to go with the extra load for now.
Thanks again,
Andreas
—
Reply to this email directly or view it on GitHub.


iOS: unknown client error adding Feedly account

When authorising a feedly account, this is returned:
{"errorCode":400,"errorId":"ap13int-sv2.2019111800.733678","errorMessage":"unknown client id"}
(Apologies if it’s not meant to be working yet.)

@brentsimmons I can send you the client id and secret for the next release. @vincode-io has them, too.

@kielgillard Please do send them to me, along with instructions for where I need to put them so that they get properly added to the app.

Is it possible to get this information for my own testing?

@davidsomen We need to keep the Feedly info very closely held. I have it only because I need to have to it in order to build the app.
Sorry about that! Obviously, in an ideal world, Feedly wouldn’t have this kind of restriction, but we respect them and their reasoning.

@brentsimmons Thank you for your response. I completely understand. I look forward to Feedly integration in the Mac app :)


Fix build on latest nightly.

We need to use the newer x86_64 crate as that fixes a compilation
error due to a rustc feature having been removed.

I will look at that CI failure in about 10 hours after u wake up again.

This should compile now. I haven't been able to run it yet as I have to do fixes to other crates along the way...

Thanks a lot!


Show number of questions yet to be answered

Two highest scoring candidates in our quiz, both had this to say:

I was fun quiz to close.
Showing the number of questions yet to be answered in the quiz would have helped me not skip a few questions. I skipped a couple of questions, I thought I was running out of time but I still had enough time to go through them.

And,

This is very interesting quiz.
One feedback is to let participant knows how many questions left and how many points left. This helps participant to decide how much time he/she should work on each question.

While I was initially against showing the number of questions, or score left, I think both options make sense to let people better manage their time. Also, with the 2 min skip blocking helps ensure that there is certain penalty associated with skipping questions -- so I feel more confident with showing the candidates how much time and score they have left.

We show the total number of questions and the current question number above the question text like
Question 1 of 29
The backend sends information about the total score left too but couldn't think of an appropriate place to display it on the frontend. I don't want to clutter the top bar. Also, I personally feel we shouldn't maybe burden the candidate with another piece of information. Maybe we can display the max possible score on the first screen. Thoughts @manishrjain?

I think each info in our top bar uses up too much horizontal space by being fancy. We can simplify all that to add more information there.
Having said that, I don't have a strong opinion -- this is something that a couple of candidates asked for. If you don't think it's helpful, feel free to skip. Just showing Q 1 of 29 etc. is sufficient for now.

I pushed out the change about showing the number of questions. I'd say let's wait sometime, if people still ask for the total score left then we'd find a way to show it.

Sounds good. Closing the issue.


Update architecture diagram

Update to streamline processing and introduce infrastructure to handle failure.

I did my best on the alignment.  Good catch on the output bucket - I simply named it Output Bucket.

Closing in favor of PR #7.


Weird FieldList behavior

At first I tried to use FieldList inside an other FieldList, and it just didn't work. So I started trying simpler things, and now I really don't understand FieldList anymore.
Can someone explain me this?
>>> from wtforms import Form, FieldList, StringField
>>> from wtforms.validators import InputRequired
>>> from werkzeug import MultiDict
>>> class Foo(Form):
...    bar = FieldList(StringField(), validators=[InputRequired()])
...
>>> # That one should work
>>> f = Foo(MultiDict([("bar", ["qux"])]))
>>> # But actually... no
>>> f.validate()
False
>>> # And this is weird
>>> f.data
{'bar': []}
>>> # Let's try it an other way
>>> class Foo(Form):
...     bar = FieldList(StringField(validators=[InputRequired()]))
...
>>> f = Foo(MultiDict([("bar", ["qux"])]))
>>> f.validate()
True
>>> # Looks good so far, let's try something bad
>>> f = Foo(MultiDict([("bar", 123)]))
>>> f.validate()
True
>>> # wait, wut?

FieldList creates multiple values with different names inside.
Try rendering the form, you'll see.
Also, please limit this for bug tracking, not questions

Okay so, I know what FieldList is meant for, and I don't think your answer help me at all. I was using WTForms for validating JSON input; rendering has nothing to do here.
I opened this issue because to my usage, this is definitely a bug, which in the end (and by the lack of any response), led me to use an other library: FieldList can't properly validate raw data like other fields.

+1. FieldList is shit!


SqlException : Incorrect syntax near the keyword 'EXCEPT'.

The error occurs when a query contains System.Linq.Queryable.Except method call
To reprocude use this test code.
using var dbContext = new AppDbContext(optionsBuilder.Options);
dbContext.Database.EnsureCreated();
var query1 = dbContext.Products.Where(x => x.Price > 100);
var query2 = dbContext.Products.Except(query1);
query2.BatchUpdate(x => new Product { Status = "Some status" });

Except is not supported for Batch, you could simple write direct oposite Query: Where(x => x.Price <= 100)

ok. I just inform you about the problem. I will use another library for my project.


🐛 [firebase_core_web] During initialization: "NoSuchMethodError: method not found: 'apply' on null"

Bug report
Describe the bug
Sometimes during initialization of a web app we see the following crash:
  NoSuchMethodError: method not found: 'apply' on null
  TypeError: Cannot read property 'apply' of undefined
  [Mapped Dart].JsObject.callMethod(third_party/dart_lang/v2/sdk/lib/_internal/js_runtime/lib/js_patch.dart:217)
  [Mapped Dart].FirebaseCoreWeb._injectSrcScript(third_party/flutter_plugins/firebase_core/web/lib/src/firebase_core_web.dart:94)
  [Mapped Dart]._wrapJsFunctionForAsync(third_party/dart_lang/v2/sdk/lib/_internal/js_runtime/lib/async_patch.dart:317)
  [Mapped Dart]._wrapJsFunctionForAsync.<anonymous>(third_party/dart_lang/v2/sdk/lib/_internal/js_runtime/lib/async_patch.dart:342)
  [Mapped Dart]._asyncStartSync(third_party/dart_lang/v2/sdk/lib/_internal/js_runtime/lib/async_patch.dart:247)
  [Mapped Dart].FirebaseCoreWeb._injectSrcScript(third_party/flutter_plugins/firebase_core/web/lib/src/firebase_core_web.dart:81)
  [Mapped Dart].FirebaseCoreWeb._injectSrcScript(third_party/flutter_plugins/firebase_core/web/lib/src/firebase_core_web.dart:81)
  [Mapped Dart].FirebaseCoreWeb._initializeCore.<anonymous>(third_party/flutter_plugins/firebase_core/web/lib/src/firebase_core_web.dart:123)
  [Mapped Dart].MappedIterator.moveNext(third_party/dart_lang/v2/sdk/lib/internal/iterable.dart:391)
  [Mapped Dart].Future.wait(third_party/dart_lang/v2/sdk/lib/async/future.dart:520)

Steps to reproduce
We're not sure how to repro this yet. We'll update as we learn more.

Flutter doctor
Click To Expand
[✓] Flutter (Channel google3, on Debian GNU/Linux rodete 5.18.16-1rodete4-amd64, locale
    en_US.UTF-8)
    • Framework revision fe9b598c12 (71 days ago), 2022-10-09T00:00:00.000
    • Engine revision ac1fec6367
    • Dart version 2f0c186d3b

[✓] Android toolchain - develop for Android devices (Android SDK version Stable)
    • Android SDK at google3
    • Platform Stable, build-tools Stable
    • Java binary at: /opt/android-studio-with-blaze-2021.3/jre/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)

[✓] Android Studio (version 4.2)
    • Android Studio at /opt/android-studio-with-blaze-4.2
    • Flutter plugin version 58.0.1
    • Dart plugin version 202.8443
    • Java version OpenJDK Runtime Environment (build 11.0.8+0-b944-P17168821)

[✓] Android Studio (version 2020.3)
    • Android Studio at /opt/android-studio-with-blaze-2020.3
    • Flutter plugin version 61.2.2
    • Dart plugin version 203.8292
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7249189)

[✓] Android Studio (version 2021.3)
    • Android Studio at /opt/android-studio-with-blaze-2021.3
    • Flutter plugin version 69.0.3
    • Dart plugin version 213.7371
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)

[✓] IntelliJ IDEA Community Edition (version 2019.2)
    • IntelliJ at /opt/intellij-ce-stable
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] VS Code (version 1.72.2)
    • VS Code at /usr/share/code
    • Flutter extension version 3.52.0

[✓] Connected device (2 available)
    • Android SDK built for x86 (mobile) • emulator-5554 • android-x86 • Android 9 (API
      28) (emulator)
    • Linux (desktop)                    • linux         • linux-x64   • Debian GNU/Linux
      rodete 5.18.16-1rodete4-amd64

• No issues found!

We've observed this error since version 1.7.0. We have not yet confirmed that this is happening after updating past version 2.0.0, but since the code path through _injectSrcScript seems to be unmodified since then I don't expect there will be a difference.
Unfortunately we don't have much more details on this crash. I will collect more information and update here as we learn more. In the meantime, do the maintainers have any suggestions for what might be going wrong here?

Thanks for the report @ChristianEdwardPadilla
Is this error occurring in debug or release mode ? Can you also provide how are you initialising Firebase ?
ex:
await Firebase.initializeApp(
    options: DefaultFirebaseOptions.currentPlatform,
  );

NoSuchMethodError: method not found: 'apply' on null
TypeError: Cannot read property 'apply' of undefined

This probably indicates a cross dependency issue or how Dart is optimizing the code, but a minimal reproducible code sample would be helpful to analyze further.

Getting the same error above

@withniyaz
If you could provide us steps to replicate along with minimal code sample, would be great to make this issue actionable.

Any update? We're also facing this issue


mat-tab-group stops audio/video player on tab leave

Reproduction
Use StackBlitz to reproduce your issue: https://stackblitz.com/fork/angular-otmu5c
Steps to reproduce:

start the audio player on the Player tab
go to the Description tab

Expected Behavior
Audio continues playing
Actual Behavior
Audio stops
Environment

Angular: 8.2.10
CDK/Material: 8.2.3
Browser(s): Chrome 77.0.3865.120
Operating System (e.g. Windows, macOS, Ubuntu): Windows 10

Description
Certainly, this is a premature optimization made at https://github.com/angular/components/blob/14c4dba1b35c52cae893dee23ecf9e4d89f0790b/src/material/tabs/tab-body.ts#L76-L91
Why are you detaching the content of the tab even when a component's user doesn't ask to?

There are also other issues concerning this detaching. For example, the view of a detached tab is inited empty and the mat-tabs-group user should implement workarounds for this.
A simple example:
there is a video player on the first tab, there is an image editor on the second tab
scenario: the user takes a snapshot from the video and it is loaded to the editor.
issue: editor can't be inited, because its tab has an empty view
workaround: hold the image editor initialization and store the snapshots until the tab view is attached.
But why do we must make such workarounds? We want to just use Tabs!

Going to track this in #19480


Poor performance when reading parquet dataset from s3

I've been facing extremely poor performance reading a parquet dataset stored in s3.
Partitioning style:  year=2020/month=10/day=18/hour=00
Compression: snappy
dataset size=40mb
The data is coming from kafka connect s3 sink, there are no _metadata files.
With debugging enabled i can see awswrangler accessing each file sequentially with significant latency.
Using pyarrow, reading the partition "day=18" takes 8 seconds;But with awswrangler it takes up to 27 seconds.
I've tried running the same script inside the VPC in an EC2 instance, however i am getting the same results. Basically what im trying to do is a batch job on the day partition and an upsert into redshift. Is reading data as parquet (snappy compression) and doing simple ETL then upserting into redshift a bad approach?
My team and I have been struggling with this issue for a while, any help or recommendation is greatly appreciated!
ps: Id rather use awswrangler because it parses the dtypes of the dataframe more accuratly, unlike pyarrow.

Hi @b00033811 !
Thanks for reaching out. I would like to understand more about your use case.

Using pyarrow, reading the partition "day=18" takes 8 seconds;But with awswrangler it takes up to 27 seconds.

Are you passing the path as s3://.../year=2020/month=10/day=18/ or are you passing s3://.../ and filtering the partitions with the partition_filter argument?
How many files do you have under the year=2020/month=10/day=18? And under the entire dataset?
40 MB are for all files sizes under year=2020/month=10/day=18 aggregated? Or is it for each file?

Is reading data as parquet (snappy compression) and doing simple ETL then upserting into redshift a bad approach?

It's a good approach, we use it all time.

ps: Id rather use awswrangler because it parses the dtypes of the dataframe more accuratly, unlike pyarrow.

Wrangler uses PyArrow under the hood only for low level operations (file level), in the dataset level we made several different choices that results in different behaviors. Each choice is a trade-off, and seems to me that you probably are in some situation where we opted to keep better data types than performance. But I don't have too much context by now, let's try to figure out an more precise answer.

Closing due to inactivity.

HI @igorborgest!
I would like to reopen this issue, as we also have seen quite unsatisfying performance using the read_parquet function. This is our setup and data below:

data is in S3, there are 1164 individual date-time prefixes under the main folder, and the total size of all files is barely 25.6 MB. So quite a lot of small individual files organized by individual prefixes by date-time
they way we gather these files is by passing path: s3://.../ and using the partition_filter. The function call looks like this:

wr.s3.read_parquet(
                path,
                dataset=True,
                partition_filter=filter(),
            )

I've run a couple of tests to verify whether there would be any speed improvement if I passed a list of prefixes for the function to combine instead of using the partition_filter but the gain was marginal. Enabling use_threads=True gave no improvement. Overall it takes around 13 minutes to collect all files... this is just too long. Downloading them with aws sync takes a few seconds.
Our main use case for operating on streams is in AWS Batch. We have some data loaders that use the data wrangler when we train our ML model in AWS Batch. We realized after some time that the main contributor to an extended training time, is the part where the data is collected from AWS using the data wrangler (primarily the wr.s3.read_parquet). Please also note that we're not taking of big data here. Most of our use cases is like described above.
At the moment we're wondering whether this can be optimized or if we should move away from the streaming approach, and simply download the data on the container for model training. Could you give some advice? What's your take on that?

Hi @konradsemsch ! Could you open a new issue, please?
This poor level of performance was never observed in our tests... We have to investigate, it is probably some corner case...

https://github.com/awslabs/aws-data-wrangler/issues/644


RuntimeError: cannot train after make_generation_fast

When I run the train.py in visualstorytelling, errors happens as list, do you have the solution?:
Traceback (most recent call last):
  File "train.py", line 109, in <module>
    main()
  File "train.py", line 105, in main
    train_model(opt, bart, dataset, test_dataset, 10)
  File "train.py", line 42, in train_model
    bart._model.train()
  File "/home/caidesheng/miniconda3/envs/teran/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1277, in train
    module.train(mode)
  File "/home/caidesheng/miniconda3/envs/teran/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1277, in train
    module.train(mode)
  File "/home/caidesheng/miniconda3/envs/teran/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1277, in train
    module.train(mode)
  File "/home/caidesheng/.cache/torch/hub/pytorch_fairseq_master/fairseq/models/fairseq_model.py", line 212, in train
    raise RuntimeError("cannot train after make_generation_fast")
RuntimeError: cannot train after make_generation_fast

it seems due to the code in fairseq_model.py of fairseq:
apply_make_generation_fast_(self, "")
def train(mode=True):
   if mode:
        raise RuntimeError("cannot train after make_generation_fast")

# this model should no longer be used for training
self.eval()
self.train = train

seems the fairseq I used is out-dated, I will turn this into huggingface as soon as possible


Config not generating properly/incorrect config version is wanted

on a fresh launch floodgate is generating the wrong config version and crashes

Please see https://github.com/GeyserMC/Floodgate-Fabric/issues/7#issuecomment-817022484 for a workaround. Thanks!


hyperconverge logstash, elasticsearch, and kibana

Signed-off-by: Kevin Carter<EMAIL_ADDRESS>

:shipit:

Thank you for submitting a pull request to this project!
Once all check jobs have passed and the pull request has received a
sufficient number of reviews, the repository's gate jobs can be triggered by
adding a comment of :shipit: to this pull request.
When the gate jobs have completed successfully, this pull request will get
merged automatically.

@cloudnull At least one reviewer has to approve the review before the shipit command will work.


operator project-quay (3.7.7)

Add v3.7.7 to community operators

/merge possible

/merge possible


removed extra break

#735

@nenonix @JohanFreelancer9 - kindly do the copy editor review for this PR. Thanks.


Error in replace function

Showed up after trying to create a graph:
└─$ yarn linear graph --project "Celo Retirements" --svg my-project.svg     1 ⨯

yarn run v1.22.19
$ bin/linear graph --project 'Celo Retirements' --svg my-project.svg
Found project 'Celo Retirements' with id e3c126f3-f085-4757-acdd-11e078166eaf
Doing issue query page 1 at start ...
  Got 50 new node(s); total now 50; hasNextPage=true
Doing issue query page 2 after 863d5910-a11c-4e63-94ba-61376ff240a2 ...
  Got 16 new node(s); total now 66; hasNextPage=false
dev/linear-analysis/src/commands/graph.ts:368
    .replaceAll('&', '&')
     ^
TypeError: s.replaceAll is not a function
    at encode (dev/linear-analysis/src/commands/graph.ts:368:6)
    at getNodeAttrs (dev/linear-analysis/src/commands/graph.ts:401:21)
    at createNode (dev/linear-analysis/src/commands/graph.ts:338:21)
    at registerNode (dev/linear-analysis/src/commands/graph.ts:358:32)
    at buildGraph (dev/linear-analysis/src/commands/graph.ts:517:5)
    at Command.run (dev/linear-analysis/src/commands/graph.ts:708:19)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

I think you're using an old node.js.  Please try nvm use stable.

This worked.  Updated README in 3d7152ea1cb843462237b5f92189f9f99c3d54c0.


Sync Fork from Upstream Repo

Pull Request Test Coverage Report for Build aef41af608864b572b8775976fdbdc9becee4e30-PR-10

67 of 67 (100.0%)  changed or added relevant lines in 15 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.3%) to 98.719%

Totals

Change from base Build 87baf5f752fe0e436ab83f863873b8657fad2ac1:
-0.3%

Covered Lines:
505

Relevant Lines:
505

💛  - Coveralls


random generation of a content of my training data

Hi all,
I'am in Linux with multi-processeurs  and i have configurated my chatette.py as follows:
from chatette.facade import Facade
import sys
facade = Facade(sys.argv[1], sys.argv[2], "rasamd", seed='ljjeek', force_overwriting=True)
facade.run()

My problem is that chatette run in // with a random manner. If i launch two times chatette.
The first Time i obtain:
intent: intent 1
intent: intent 2
intent: intent 3
The second time i obtain:
intent: intent 3
intent: intent 1
intent: intent 2
So the order of the intents is not the same. Is there a solution to fix this issue ?
Thanks

Hello,
Thanks for pointing this out.
It looks like a bug, but I am unable to reproduce it: running your code and using the airport example as a template always generates an output with the intents in the same order.
Would you mind posting the template file you used here?
Also, could you explain why it's important for you to have intents always generated in the same order?
Thanks in advance :)

I'll close this since I got no response and can't reproduce it.
Feel free to reopen it if you can give me more information.


Improve optional libraries install messages

Summary
Fixes #1181
Details and comments

I liked @Cryoris  idea of having a special exception class for it. It is a more elegant solution. I will create one.


Can I use needs web scrapping/crawling

Can I use is not actively maintained enough, the number of features where it has outdated informations, especially concerning chrome/blink feature supports, is astonishing.
But it happen that chromium features support has already an always up to date, complete knowledge base.
It's called https://chromestatus.com/features
We just needs to scrap each chromestatus feature
And get:
The feature title
The support status and the corresponding version.
Other data is bonus.
Then go to can I use matching feature title
And overwrite chrome data (and eventuality opera, new edge, etc)
The difficulty is the mapping between chromestatus feature title and can I use feature title.
So manually we should contribute
Association pairs:
{chrome status title, can I use title}
E.g :
{"CSS Scroll Snap Points", "CSS Scroll snap" }
So yes there is a quick manual effort but you do it once and then data will be updated automatically (just periodically re run the scrapper) which will be a huge improvement for showing what we can really use.
Bonus 1:
Scrap other browsers feature supports on can I use.
Disclaimer I don't know how much maintained it is for other browsers.
Bonus 2:
Many chromestatus features have no equivalent in can I use because can I use is deeply incomplete.
Then a contributor could just give an association key
{chromestatus title, NULL}
Which would then automatically create a new can I use feature using the same name, description and data as chromestatus.
Doing this would help make the world a better place.
I wonder if you could propose this as a Gsoc (Google summer of code)
BTW instead of scrapping, maybe they have an API?

Regarding maintenance of the caniuse data, it's partly because of this that recently the MDN Browser Compatibility Data was included in caniuse searches. This data is more actively maintained than the caniuse data and aims to cover all web technologies, where caniuse data prioritizes features that have the most developer interest.
Including the data provided by the Chrome Platform Status project is interesting and worth considering, however the biggest problem I see is that their data focuses primarily on the status in Chromium rather than across all of 19 browsers tracked by caniuse. Yes, they track if a feature is "shipped" in a few other browsers, but doesn't provide the version in which it was shipped or any other details. Most of the work in getting compatibility data is doing the testing required across browsers so that work would still need to be done.
Another concern I have is that because the Chrome Platform Status project is focused primarily on features being worked on in Chrome/Chromium I feel this would unfairly skew the data found on caniuse towards Chrome features compared to those being developed in other browsers.
I'll leave this issue open because I think it could still be of value to include CPS results in search results similar to what's happening with the BCD data, though as mentioned since the data's much more limited it would probably just link to the CPS site rather than provide a support table.

Chrome Status is reliable mostly because for the last several years I've hounded feature owners to keep it up to date. I would't build a system with a single human point of failure.
Regarding MDN, we (Google) are trying to make Chrome data more reliable. I can't say more than that now.

I was thinking about importing Chrome Status data into BCD, but never really looked into it more closely. If anyone is interested in doing that, let me know :)
I agree Chrome Status is Chrome biased. It even says "After a feature ships in Chrome, the values listed here are not guaranteed to be up to date." So, it happens to be wrong about other browser statuses in many entries, so I believe the site could just stop talking about non-Chrome implementations and refer to BCD/caniuse instead.


Method to determine if OTA task has checked for a queued update

Is your feature request related to a problem? Please describe.
We are using OTA with a battery powered device which only connects periodically to Wifi. We would like to connect, check for a pending OTA update, perform the update if a job is queued or disconnect as soon as possible if there is no job queued. Currently the OTA_Agent.h interface does not provide a means to determine if the check for a queued job has completed.
The existing OTA_CheckForUpdate function sounds like it should do this, but it seems to just publish a message, not actually check the result?
Describe the solution you would like.
A callback event in OTA_JobEvent_t to indicate the a check has copmleted or fix the implementation of OTA_CheckForUpdate so that it only returns on response from the server and actually indicates whether there is a job or not.
Additional context
Thank you!

Hi @thealastair,
Many customers have made this feature request -- we're working on improving the OTA update agent to accommodate it.

Hi Gordon, thanks for the reply, is there also a new OTA interface coming which uses the new MQTT client? In the feature/ble-beta branch I see there is an implementation of the existing OTA_Agent.h interface using the new client, but I thought there might a change to the interface coming as well?

Hi @thealastair, we may be changing the interface ahead of publishing the ble-beta work, which uses the latest libraries, but we are not sure yet. Is there any particular reason why you may want to use the new MQTT lib with OTA? I am interested in understanding your requirements fully, as we may re-design OTA a bit. So far I have:

Ability to check for pending updates (notification)
...

thank you for your feedback!

We're using the AWS shadow client as well as OTA and currently this requires two MQTT connections (because the shadow client implementation doesn't allow sharing) which causes resource problems on our platform, so we'd like to migrate to the new IOT framework which I believe supports shared connections.

Hi Alastair,
Is this issue resolved or are you still facing any problems?

There's doesn't appear to have been any change to the API (at least in the master branch), so no, this issue is not resolved.

Hi, has there been any progress made on this?

Hi,
We are working on providing an example for this and will update here when it is merged in master branch.

We are still actively working on this and will merge the fix to master in coming releases.

@qiutongs This doesn't yet seem to be supported, why was the issue closed? If it is supported, is there an example of how to implement this?

@pvyawaha or @qiutongs is there any news on this feature?
Are there any examples of doing OTA while operating on battery power?


Change package name

Current package name is: me.ivmg.*
Change to something like: com.kotlintelegrambot.*

I will take on this once #44 is merged since it will break a lot of things.

#52 has been merged, you can proceed whenever you want


[Production & Staging] Dataverse: Deleting a Published version of a file is allowed, but not really.

Steps

Go to a project with dataverse configured
Click to view a published file that is also in draft version
Click on the delete button the files detail page
Confirm that you want to delete the file
Return to the Files tab
Look for the file in the grid under Published files

Expected
Completing the steps to delete the published version of a file should not be allowed since those are read-only. Returning to the file tab and seeing the file remain when the steps for deleting were completed with no errors.
Actual
By completing the steps with no errors, the user is led to believe that they in fact, deleted a published file.  However, they only deleted the draft version of the file.

@nicipfeiffer After the steps above, is the Published file still visible? But not any longer the Draft file?

@lbanner Yes, the published file is still visible.  The draft file is now deleted.  The inconsistency, in my opinion, is that the user is completing the deleting action from the published file.  I think this is misleading.

@rliebz Did your resolving commit here get merged into Production? Thanks.

I'm bundling the commit with a few other changes, which includes a better distinction between draft/published files and some related permissions fixes. Still a WIP.

OK cool :+1:

@lbanner @GaryKriebel This has regressed. I am seeing a Delete button on the files detail page for my published Dataverse files. I think we need to reopen this.

Ugh - got it.

@caileyfitz can you have a QA team member replicate to verify? Thanks.

@lbanner and @caileyfitz I am also seeing the delete button on the files detail page.


query history?

is there any?
and also - how to backspace text in query? bs jumps to conn list

Hi,
query history is planned, i'm focusing on solving important bugs first. PRs are welcome.
backspace goes to connection list only if a text view is not focused, so when have focus on filter input or query editor, it should work properly, i just tested and it works fine.


Add support for <!-- --> comments in README.mds

In Azure SDKs are trying to start embedding README.md. One issue we've hit is that HTML-style comments that github supports (e.g. <!-- This is a comment -->) are rendered directly and not committed.
It would be great if support for stripping comments was added to nuget.org gallery

@pakrym , thanks for the suggestion! If you are interested in contributing a PR we will gladly work with you to merge and ship the change.