Add support for PartitionBy functionality

Which issue does this PR close?
Closes #1404.
Rationale for this change
DataFusion lacks support for partition by operation.
The most used example is" given a dataset, we need to write it down on the storage as a partitioned set of files (ie: Parquet dataset partitioned by year/month/day, etc). We need to write it as paths like this:
/dataset/day=2021-12-28/fuel=Gas/
/dataset/day=2021-12-28/fuel=Diesel/
/dataset/day=2021-12-27/fuel=Electric/
...

The pattern above is not supported and the collect_partitioned method returns data only hash partitioned.
What changes are included in this PR?
This PR adds a new partitioning method: Partitioning::PartitionBy.
It has two parameters:

the expression by which the partitioning will take place
an optional value to specify the number of partition to output:

the best would be to know the exact number of distinct values by which the data will be partitioned, fact that will be beneficial in terms of performance
if it Is bigger than the  exact number of distinct values for partitioning it will return the number of partitions found in the data
if is smaller it will return the first n number of partitions dropping the other
if not specified, it will start from i16::MAX

Are there any user-facing changes?
There is a new partitioning option available as  part of the API.

Thanks for showing interest in this functionality!
I believe this functionality shouldn't be a part of the RepartitionExec. The partitions (confusingly) mentioned there are a unit of parallelism and distributing data, not a way to aggregate data like in the example for writing to different partitions. The partitions in PartitionExec are equivalent to Spark partitions and mirror that functionality / design.
We already have some pieces that we could use to implement the write support:

hash aggregation, i.e. group by. This could be used to group the same rows and append them to the output directories / files. HashAggregateExec has different modes, local and global. For writing, only local mode is needed, as we can write the rows to multiple files within the same directory (just like Spark does).
RepartitionExec::Hash, allowing the parallelize the operations on more CPUs and nodes in Ballista. This is already utilized by using the hash aggregate code.

The missing piece here is the support/wiring to write the final partitions to directories / files using a partitioning scheme.
I don't have to much time ATM to write up a proposal myself, but if you see which direction I am going to, I support to write a design for this to collect feedback on it.
It would be wise to look after some similar engines, like Spark/Hive/Trino/etc. for some inspiration.

@Dandandan Thanks for the message. I didn't have any other way of reading data and writing it partitioned to the storage. If you have such example please guide me to it.
My use case is this: read parquet data (multiple files) re-partition them and write them as OSS Delta. In my case I read the data in memory and collect it as partitioned then write each partition as separate file.
Can you give me an example using DataFrame API of writing a data as parquet? Or if there is a section where I can implement the Delta writer or any other writer?
I do not agree with the fact that this useful only for the write moment and neither going towards grouping and aggregating is a good way. I'm thinking of the following reasons:

When using group by it requires an aggregation function and this is not what it is needed. We don't aggregate anything we just shuffle the data in a specific way.
I've seen that the shuffling process is implemented nicely using channels (for the other repartitioning modes) and I think that the same approach has to be used in this case too.
Parallel processing is still needed after partitioning the data in this way. Summarising metrics or doing some aggregations  over such partitioned data is easy parallelizable which is a benefit.
Using aggregation instead of shuffling will incur more performance penalty due to the fact that it needs to apply aggregation and keep some state.

There are other benefits or pros that this partitioning brings in but in terms of cons I cannot think of too many and no blocker (I have limited knowledge of DataFusion at this moment). Some that come to my mind are these:

In some cases there is a more unbalanced partitioning trend which is ok. In the case of Partitioning::Hash the unbalanced partitioning happens too. So I don't know if this is really a downside.
A bit slower than the Partitioning::Hash due to locking.

If there are any other cons that I'm not aware of, please explain them to me. Those will improve my knowledge on DataFusion.
If the name the I did give to this functionality is not good - let's say that PartitionBy term is used for writing and that is misleading - then we can say something else like ShuffleByExpression or something else.

Marking PRs without activity in the last month as stale. I'll plan to close it in another month or so without activity, though feel free to reopen it when you have time to work on it)

Closing stale PRs. Please reopen (or open a new one) if you plan to keep working on this feature.

@alamb How can I reopen this PR? I don't see any re-open PR button. This is not a stale PR is just waiting feedback from your side.

@alamb, @houqp: I don't know why this PR was considered stalled and closed when the last comment was mine and this PR is waiting feedback from your side. I don't know how to reopen the PR as I see no button to do that.

I can help take a look into this next week and answer some of your questions if @Dandandan is busy with other things

I can help take a look into this next week and answer some of your questions if @Dandandan is busy with other things

Thanks @houqp !

Thanks @andrei-ionescu  -- I am sorry I missed that this PR was waiting on feedback from us; Thank you @houqp  for taking that on!

I believe that PR is the way forward and addresses my concerns I raised here earlier.

Marking as draft as it is still in progress

Closing for now to clean up PR list -- feel free to reopen to work on it more


cache XHRs (like npm package.json) locally so subsequent build runs don't refetch them

Don't think we can do this because the XHR results may change (e.g., unpkg version, number of stars, metadata).


SEO: Culture Code is not assigned to Custom URL

Description of bug
I needed to create a Friendly URL for S.E.O. and I used the S.E.O. configurations existing in all pages.
When I create this Friendly URL in Default Language (en-US) it works the way it should work but when I create in other language, it simple don't work.
I follow this instructions and don't work in 'pt-PT'.
Steps to reproduce

Website localised with 'pt-PT' and 'en-US' languages;
In pt-PT language, Go to 'Content, Pages, Advanced, S.E.O.';
Chose a Page and Click on 'Add URL' and add 'URL Path';
Save the changes.

Current result
In DB table TabURLS, the 'Culture Code' for the custom URL is 'en-US' not for example 'pt-PT'.
Expected result
Is expected the 'Culture Code' to use the Current Language (pt-PT), not the Default Language.
Affected version

[x] 9.2.1 (only version tested)

Affected browser

Not browser related

@OVigilante can you confirm if this is still an issue?  Thanks.

I confirm it is still an issue in 9.6.2


Kafka not working after upgrading to Hoxton.SR9

Hello,
I have been getting the following exception in all our services that include Kafka after upgrading to Hoxton.SR9.
With Hoxton.SR8 is working just fine.
org.springframework.kafka.listener.ListenerExecutionFailedException: Listener failed; nested exception is org.springframework.messaging.MessageDeliveryException:
Somehow the service is not able to subscribe to the kafka topic.
We are also using Spring Boot 2.3.6.RELEASE
Any idea what could be triggering this error?
Thank you in advance!

You need to show the full stack trace.

Stacktrace:
.springframework.kafka.KafkaException: Seek to current after exception; nested exception is org.springframework.kafka.listener.ListenerExecutionFailedException: Listener failed; nested exception is org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel 'classification-group-provider-1.mapClassificationGroup-in-0'.; nested exception is org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers, failedMessage=GenericMessage [payload=byte[6129], headers={kafka_offset=1249928, scst_nativeHeadersPresent=true, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@52f30b66, deliveryAttempt=3, kafka_timestampType=CREATE_TIME, kafka_receivedPartitionId=61, kafka_receivedMessageKey=1001799931, contentType=application/json, kafka_receivedTopic=product-updates, kafka_receivedTimestamp=1605146692343, kafka_groupId=classification-group-provider, accept=application/json}], failedMessage=GenericMessage [payload=byte[6129], headers={kafka_offset=1249928, scst_nativeHeadersPresent=true, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@52f30b66, deliveryAttempt=3, kafka_timestampType=CREATE_TIME, kafka_receivedPartitionId=61, kafka_receivedMessageKey=1001799931, contentType=application/json, kafka_receivedTopic=product-updates, kafka_receivedTimestamp=1605146692343, kafka_groupId=classification-group-provider, accept=application/json}]

Caused by: org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel 'classification-group-provider-1.mapClassificationGroup-in-0'

This is a spring-cloud-stream issue; please open it in that project; since it's in a different org, I can't move this there.
Show your configuration (application.yml).

Thank you for your answer, I opened the issue there just before I did it here and they told me I had to open it here.

Right; you should always include as much information as possible, including a stack trace, to avoid such time wasting. With the simple description, I don't blame them for sending you here.

Alright, i just posted the issue there with everything they could need. Thank you very much for your time.


[Mobile] Audit: Select

Description
Audit the use cases and requirements for Select  component
Use the FigJam to collate screenshots, notes on component api and requirements etc.
FigJam : https://www.figma.com/file/ARJUQovVqX6QkGgiIpPOD3/Picker%2FSelect-Audit?node-id=0%3A1&t=srUuLIOpFDUOvCrv-1
Technical Details

Collect existing use cases across your assigned platform
Collect similar examples from third party design systems
Note down behaviours, attributes, questions and requirements that you notice about these use cases

Acceptance Criteria

The majority if not all use cases from your assigned platform have been collected in screenshots and added to the FigJam
At least 3 examples from third party design systems have been collected and added to the FigJam
Have listed possible component names and identified your preferred name based on your research
Have listed down possible component api and identified the options

References

FigJam
Read exercised #05 Identify Existing Paradigms in Design and Code and #06 Identify Emerging and Interesting Paradigms in Design and Code in the Design System in 90 Days workbook

This was completed in the Picker/Select audit https://github.com/MetaMask/design-tokens/issues/358
https://www.figma.com/file/ARJUQovVqX6QkGgiIpPOD3/Picker%2FSelect%2FDropdown-Audit?type=whiteboard&node-id=0%3A1&t=IHxtsX59Di2cnAyj-1


SVM: use scikit-learn instead of LIBSVM

Separate changes out from quire branch.

One thing to be concern is that changing from LIBSVM to scikit-learn is that scikit-learn only has the interface for predict_proba, which will take relatively long to train in compare with calculating the distance to margin (which predict_real only needs).
@hsuantien
Should we still switch scikit-learn or I should just try to calculate the margin with primal coefficient?

From scikit-learn document
The SVC
http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC
method decision_function gives per-class scores for each sample (or a
single score per sample in the binary case).
So I think it is possible to just connect our predict_real to
decision_function?
Thanks.
--HT
On Wed, Dec 16, 2015 at 1:28 PM, yangarbiter<EMAIL_ADDRESS>wrote:

One thing to be concern is that changing from LIBSVM to scikit-learn is
that scikit-learn only has the interface for predict_proba, which will take
relatively long to train in compare with calculating the distance to margin
(which predict_real only needs).
@hsuantien https://github.com/hsuantien
Should we still switch scikit-learn or I should just try to calculate the
margin with primal coefficient?
—
Reply to this email directly or view it on GitHub
https://github.com/ntucllab/libact/issues/15#issuecomment-164998494.

--
Hsuan-Tien Lin<EMAIL_ADDRESS>http://www.csie.ntu.edu.tw/~htlin
Associate Professor
Dept. of Computer Science and Information Engineering
& Graduate Institute of Networking and Multimedia
National Taiwan University

I've also change logistic_regression's predict_prob to decision_function too.

Will this affect the outcome of QSs?

For uncertainty sampling, theoretically it would only affect the method least entropy (which I took it out int #1)

#19 merged. Please refer to #21 for the predict_real issue.


Conversion of acceptance tests to pytest

This PR converts the remaining acceptance tests from BATS to pytest.
This is a large pull request, however most of the changes are fairly mechanical conversion from existing BATS tests. Reviewers should focus on the changes in bin/improver-tests and lib/improver/tests/acceptance/acceptance.py as these apply across all pytest-based acceptance tests, including future work by the team generally.
There are currently 3 tests which xfail - these should be fixed in separate pull requests as some thought will be required how to handle them.
The BATS tests are not deleted in this pull request, but the improver tests CLI is converted to use pytest for both unit and acceptance tests.
Testing:

[x] Ran tests and they passed OK
[x] Added new tests for the new feature(s)

 Here is an overview of what got changed by this pull request:

Issues
======
- Added 52

Complexity increasing per file
==============================
- lib/improver/tests/acceptance/test_cli_apply_lapse.py  1
- lib/improver/tests/acceptance/test_cli_wind_downscaling.py  3
- lib/improver/tests/acceptance/test_cli_combine.py  1
- lib/improver/tests/acceptance/test_cli_nowcast_extrapolate.py  2
- lib/improver/tests/acceptance/test_cli_snow_falling_level.py  2
- lib/improver/tests/acceptance/test_cli_weighted_blending.py  2
- lib/improver/tests/acceptance/test_cli_wind_gust_diagnostic.py  1
- lib/improver/tests/acceptance/test_cli_wxcode.py  3
- lib/improver/tests/acceptance/test_cli_update_grid_metadata.py  2
- lib/improver/tests/acceptance/test_cli_extend_radar_mask.py  1
- lib/improver/tests/acceptance/test_cli_percentile.py  1
- lib/improver/tests/acceptance/acceptance.py  8
- lib/improver/tests/acceptance/test_cli_spot_extract.py  3
- lib/improver/tests/acceptance/test_cli_nbhood.py  1
- lib/improver/tests/acceptance/test_cli_standardise.py  2
- lib/improver/tests/acceptance/test_cli_nbhood_iterate.py  3
- lib/improver/tests/acceptance/test_cli_percentiles_to_realizations.py  3
- lib/improver/tests/acceptance/test_cli_extract.py  2
- lib/improver/tests/acceptance/test_cli_recursive_filter.py  1
- lib/improver/tests/acceptance/test_cli_feels_like_temp.py  2
- lib/improver/tests/acceptance/test_cli_estimate_emos.py  2
- lib/improver/tests/acceptance/test_cli_nbhood_landsea.py  3
- lib/improver/tests/acceptance/test_cli_temporal_interpolate.py  5
- lib/improver/tests/acceptance/test_cli_nowcast_optical_flow.py  3
- lib/improver/tests/acceptance/test_cli_wind_direction.py  1
- lib/improver/tests/acceptance/test_cli_temp_lapse_rate.py  2
- lib/improver/tests/acceptance/test_cli_time_lagged_ensembles.py  2
- lib/improver/tests/acceptance/test_cli_percentiles_to_probabilities.py  1
- lib/improver/tests/acceptance/test_cli_blend_adjacent_points.py  2
- lib/improver/tests/acceptance/test_cli_probabilities_to_realizations.py  2
- lib/improver/tests/acceptance/test_cli_create_halo.py  1
- lib/improver/tests/acceptance/test_cli_neighbour_finding.py  2
- lib/improver/tests/acceptance/test_cli_generate_topography_bands.py  2
- lib/improver/tests/acceptance/test_cli_between_thresholds.py  1
- lib/improver/tests/acceptance/test_cli_wet_bulb_temperature.py  2
- lib/improver/tests/acceptance/test_cli_generate_landmask_ancillary.py  1
- lib/improver/tests/conftest.py  1
- lib/improver/tests/acceptance/test_cli_resolve_wind_components.py  1
- lib/improver/tests/acceptance/test_cli_apply_emos.py  2
- lib/improver/tests/acceptance/test_cli_orographic_enhancement.py  3
- lib/improver/tests/acceptance/test_cli_threshold.py  1

Clones added
============
- lib/improver/tests/acceptance/test_cli_wind_downscaling.py  4
- lib/improver/tests/acceptance/test_cli_nowcast_extrapolate.py  2
- lib/improver/tests/acceptance/test_cli_percentile.py  2
- lib/improver/tests/acceptance/test_cli_standardise.py  2
- lib/improver/tests/acceptance/test_cli_estimate_emos.py  2
- lib/improver/tests/acceptance/test_cli_nbhood_landsea.py  2
- lib/improver/tests/acceptance/test_cli_nowcast_optical_flow.py  2
- lib/improver/tests/acceptance/test_cli_apply_emos.py  2
- lib/improver/tests/acceptance/test_cli_orographic_enhancement.py  2

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Complexity increasing per file
==============================
- lib/improver/tests/acceptance/test_cli_combine.py  1
- lib/improver/tests/acceptance/test_cli_feels_like_temp.py  2
- lib/improver/tests/acceptance/test_cli_spot_extract.py  3
- lib/improver/tests/acceptance/test_cli_recursive_filter.py  1
- lib/improver/tests/acceptance/test_cli_nowcast_optical_flow.py  3
- lib/improver/tests/acceptance/test_cli_wind_direction.py  1
- lib/improver/tests/acceptance/acceptance.py  8
- lib/improver/tests/acceptance/test_cli_nbhood_iterate.py  3
- lib/improver/tests/acceptance/test_cli_standardise.py  2
- lib/improver/tests/acceptance/test_cli_apply_lapse.py  1
- lib/improver/tests/acceptance/test_cli_extend_radar_mask.py  1
- lib/improver/tests/acceptance/test_cli_wet_bulb_temperature.py  2
- lib/improver/tests/acceptance/test_cli_resolve_wind_components.py  1
- lib/improver/tests/acceptance/test_cli_probabilities_to_realizations.py  2
- lib/improver/tests/acceptance/test_cli_snow_falling_level.py  2
- lib/improver/tests/acceptance/test_cli_orographic_enhancement.py  3
- lib/improver/tests/acceptance/test_cli_wxcode.py  3
- lib/improver/tests/conftest.py  1
- lib/improver/tests/acceptance/test_cli_weighted_blending.py  2
- lib/improver/tests/acceptance/test_cli_neighbour_finding.py  2
- lib/improver/tests/acceptance/test_cli_generate_landmask_ancillary.py  1
- lib/improver/tests/acceptance/test_cli_blend_adjacent_points.py  2
- lib/improver/tests/acceptance/test_cli_generate_topography_bands.py  2
- lib/improver/tests/acceptance/test_cli_extract.py  2
- lib/improver/tests/acceptance/test_cli_nowcast_extrapolate.py  2
- lib/improver/tests/acceptance/test_cli_nbhood.py  1
- lib/improver/tests/acceptance/test_cli_percentile.py  1
- lib/improver/tests/acceptance/test_cli_time_lagged_ensembles.py  2
- lib/improver/tests/acceptance/test_cli_threshold.py  1
- lib/improver/tests/acceptance/test_cli_wind_downscaling.py  3
- lib/improver/tests/acceptance/test_cli_temporal_interpolate.py  5
- lib/improver/tests/acceptance/test_cli_apply_emos.py  2
- lib/improver/tests/acceptance/test_cli_estimate_emos.py  2
- lib/improver/tests/acceptance/test_cli_between_thresholds.py  1
- lib/improver/tests/acceptance/test_cli_nbhood_landsea.py  3
- lib/improver/tests/acceptance/test_cli_update_grid_metadata.py  2
- lib/improver/tests/acceptance/test_cli_create_halo.py  1
- lib/improver/tests/acceptance/test_cli_temp_lapse_rate.py  2
- lib/improver/tests/acceptance/test_cli_percentiles_to_realizations.py  3
- lib/improver/tests/acceptance/test_cli_percentiles_to_probabilities.py  1
- lib/improver/tests/acceptance/test_cli_wind_gust_diagnostic.py  1

Clones added
============
- lib/improver/tests/acceptance/test_cli_nowcast_optical_flow.py  2
- lib/improver/tests/acceptance/test_cli_standardise.py  2
- lib/improver/tests/acceptance/test_cli_orographic_enhancement.py  2
- lib/improver/tests/acceptance/test_cli_nowcast_extrapolate.py  2
- lib/improver/tests/acceptance/test_cli_percentile.py  2
- lib/improver/tests/acceptance/test_cli_wind_downscaling.py  4
- lib/improver/tests/acceptance/test_cli_apply_emos.py  2
- lib/improver/tests/acceptance/test_cli_estimate_emos.py  2
- lib/improver/tests/acceptance/test_cli_nbhood_landsea.py  2

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Complexity increasing per file
==============================
- lib/improver/tests/acceptance/test_cli_wet_bulb_temperature.py  2
- lib/improver/tests/acceptance/test_cli_wind_downscaling.py  3
- lib/improver/tests/acceptance/acceptance.py  8
- lib/improver/tests/acceptance/test_cli_recursive_filter.py  1
- lib/improver/tests/acceptance/test_cli_generate_landmask_ancillary.py  1
- lib/improver/tests/acceptance/test_cli_neighbour_finding.py  2
- lib/improver/tests/acceptance/test_cli_apply_lapse.py  1
- lib/improver/tests/acceptance/test_cli_blend_adjacent_points.py  2
- lib/improver/tests/acceptance/test_cli_feels_like_temp.py  2
- lib/improver/tests/acceptance/test_cli_nbhood_landsea.py  3
- lib/improver/tests/acceptance/test_cli_percentiles_to_realizations.py  3
- lib/improver/tests/acceptance/test_cli_percentiles_to_probabilities.py  1
- lib/improver/tests/acceptance/test_cli_nowcast_extrapolate.py  2
- lib/improver/tests/conftest.py  1
- lib/improver/tests/acceptance/test_cli_weighted_blending.py  2
- lib/improver/tests/acceptance/test_cli_apply_emos.py  2
- lib/improver/tests/acceptance/test_cli_snow_falling_level.py  2
- lib/improver/tests/acceptance/test_cli_resolve_wind_components.py  1
- lib/improver/tests/acceptance/test_cli_between_thresholds.py  1
- lib/improver/tests/acceptance/test_cli_percentile.py  1
- lib/improver/tests/acceptance/test_cli_orographic_enhancement.py  3
- lib/improver/tests/acceptance/test_cli_update_grid_metadata.py  2
- lib/improver/tests/acceptance/test_cli_estimate_emos.py  2
- lib/improver/tests/acceptance/test_cli_extend_radar_mask.py  1
- lib/improver/tests/acceptance/test_cli_temporal_interpolate.py  5
- lib/improver/tests/acceptance/test_cli_time_lagged_ensembles.py  2
- lib/improver/tests/acceptance/test_cli_temp_lapse_rate.py  2
- lib/improver/tests/acceptance/test_cli_nbhood_iterate.py  3
- lib/improver/tests/acceptance/test_cli_wxcode.py  3
- lib/improver/tests/acceptance/test_cli_spot_extract.py  3
- lib/improver/tests/acceptance/test_cli_nowcast_optical_flow.py  3
- lib/improver/tests/acceptance/test_cli_wind_direction.py  1
- lib/improver/tests/acceptance/test_cli_generate_topography_bands.py  2
- lib/improver/tests/acceptance/test_cli_nbhood.py  1
- lib/improver/tests/acceptance/test_cli_combine.py  1
- lib/improver/tests/acceptance/test_cli_create_halo.py  1
- lib/improver/tests/acceptance/test_cli_standardise.py  2
- lib/improver/tests/acceptance/test_cli_threshold.py  1
- lib/improver/tests/acceptance/test_cli_wind_gust_diagnostic.py  1
- lib/improver/tests/acceptance/test_cli_extract.py  2
- lib/improver/tests/acceptance/test_cli_probabilities_to_realizations.py  2

Clones added
============
- lib/improver/tests/acceptance/test_cli_wind_downscaling.py  4
- lib/improver/tests/acceptance/test_cli_nbhood_landsea.py  2
- lib/improver/tests/acceptance/test_cli_nowcast_extrapolate.py  2
- lib/improver/tests/acceptance/test_cli_apply_emos.py  2
- lib/improver/tests/acceptance/test_cli_percentile.py  2
- lib/improver/tests/acceptance/test_cli_orographic_enhancement.py  2
- lib/improver/tests/acceptance/test_cli_estimate_emos.py  2
- lib/improver/tests/acceptance/test_cli_nowcast_optical_flow.py  2
- lib/improver/tests/acceptance/test_cli_standardise.py  2

See the complete overview on Codacy

@tjtg @arh89 @TomekTrzeciak
Please can you not merge this without letting me know. We have a few CLI changes in progress which may cause conflicts.

@tjtg @arh89 @TomekTrzeciak
I've now merged the outstanding blue team work that could conflict with this PR. I've agreed with Tom that the rest of our outstanding work will go in after this PR.

@tjtg, this is how we could possibly use improver --verbose instead of stringify
diff --git a/lib/improver/cli/__init__.py b/lib/improver/cli/__init__.py
index ebd940cd..3f640392 100644
--- a/lib/improver/cli/__init__.py
+++ b/lib/improver/cli/__init__.py
@@ -31,6 +31,7 @@
 """init for cli and clize"""

 from collections import OrderedDict
+import pathlib

 import clize
 from clize import parameters
@@ -356,18 +357,28 @@ def execute_command(dispatcher, prog_name, *args,
             # process nested commands recursively
             arg = execute_command(dispatcher, prog_name, *arg,
                                   verbose=verbose, dry_run=dry_run)
-        if not isinstance(arg, str):
+        if isinstance(arg, pathlib.PurePath):
+            arg = str(arg)
+        elif not isinstance(arg, str):
             arg = ObjectAsStr(arg)
         args[i] = arg

+    exc = None
+
     if dry_run:
         result = args
     else:
-        result = dispatcher(prog_name, *args)
+        try:
+            result = dispatcher(prog_name, *args)
+        except Exception as exc:
+            result = exc

     if verbose or dry_run:
         print(prog_name, *args, ' -> ', ObjectAsStr.obj_to_name(result))

+    if exc is not None:
+        raise exc
+
     return result

 Here is an overview of what got changed by this pull request:

Issues
======
- Added 2

Complexity increasing per file
==============================
- lib/improver/tests/acceptance/test_neighbour_finding.py  2
- lib/improver/tests/acceptance/test_uv_index.py  2
- lib/improver/tests/acceptance/test_wet_bulb_temperature.py  2
- lib/improver/tests/acceptance/test_recursive_filter.py  1
- lib/improver/tests/acceptance/test_weighted_blending.py  2
- lib/improver/tests/acceptance/test_percentile.py  1
- lib/improver/tests/acceptance/test_feels_like_temp.py  2
- lib/improver/tests/acceptance/test_temporal_interpolate.py  5
- lib/improver/tests/acceptance/test_resolve_wind_components.py  1
- lib/improver/tests/acceptance/test_apply_emos_coefficients.py  2
- lib/improver/cli/__init__.py  4
- lib/improver/tests/acceptance/test_threshold.py  1
- lib/improver/tests/acceptance/test_nbhood_iterate_with_mask.py  3
- lib/improver/tests/acceptance/test_nbhood.py  1
- lib/improver/tests/acceptance/test_generate_landmask_ancillary.py  1
- lib/improver/tests/acceptance/test_standardise.py  2
- lib/improver/tests/acceptance/test_probabilities_to_realizations.py  2
- lib/improver/tests/acceptance/acceptance.py  9
- lib/improver/tests/acceptance/test_apply_lapse_rate.py  1
- lib/improver/tests/acceptance/test_between_thresholds.py  1
- lib/improver/tests/acceptance/test_estimate_emos_coefficients.py  2
- lib/improver/tests/acceptance/test_create_grid_with_halo.py  1
- lib/improver/tests/acceptance/test_update_grid_metadata.py  2
- lib/improver/tests/acceptance/test_temp_lapse_rate.py  2
- lib/improver/tests/acceptance/test_snow_falling_level.py  2
- lib/improver/tests/acceptance/test_nowcast_optical_flow.py  3
- lib/improver/tests/acceptance/test_extract.py  2
- lib/improver/tests/acceptance/test_blend_adjacent_points.py  2
- lib/improver/tests/acceptance/test_wind_direction.py  1
- lib/improver/tests/acceptance/test_spot_extract.py  3
- lib/improver/tests/acceptance/test_combine.py  1
- lib/improver/tests/acceptance/test_wxcode.py  3
- lib/improver/tests/acceptance/test_percentiles_to_realizations.py  3
- lib/improver/tests/acceptance/test_nowcast_extrapolate.py  2
- lib/improver/tests/conftest.py  1
- lib/improver/tests/acceptance/test_orographic_enhancement.py  3
- lib/improver/tests/acceptance/test_time_lagged_ensembles.py  2
- lib/improver/tests/acceptance/test_wind_gust_diagnostic.py  1
- lib/improver/tests/acceptance/test_generate_topography_bands.py  2
- lib/improver/tests/acceptance/test_wind_downscaling.py  3
- lib/improver/tests/acceptance/test_extend_radar_mask.py  1
- lib/improver/tests/acceptance/test_percentiles_to_probabilities.py  1
- lib/improver/tests/acceptance/test_nbhood_land_and_sea.py  3

Clones added
============
- lib/improver/tests/acceptance/test_percentile.py  2
- lib/improver/tests/acceptance/test_apply_emos_coefficients.py  2
- lib/improver/tests/acceptance/test_nbhood_iterate_with_mask.py  1
- lib/improver/tests/acceptance/test_standardise.py  2
- lib/improver/tests/acceptance/test_estimate_emos_coefficients.py  2
- lib/improver/tests/acceptance/test_snow_falling_level.py  1
- lib/improver/tests/acceptance/test_nowcast_optical_flow.py  2
- lib/improver/tests/acceptance/test_wind_direction.py  1
- lib/improver/tests/acceptance/test_wxcode.py  1
- lib/improver/tests/acceptance/test_nowcast_extrapolate.py  2
- lib/improver/tests/acceptance/test_orographic_enhancement.py  2
- lib/improver/tests/acceptance/test_wind_downscaling.py  5

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Issues
======
- Added 2

Complexity increasing per file
==============================
- lib/improver/tests/acceptance/test_extract.py  2
- lib/improver/tests/acceptance/test_between_thresholds.py  1
- lib/improver/tests/acceptance/test_time_lagged_ensembles.py  2
- lib/improver/tests/acceptance/test_apply_lapse_rate.py  1
- lib/improver/tests/acceptance/test_nowcast_extrapolate.py  2
- lib/improver/cli/__init__.py  4
- lib/improver/tests/acceptance/test_extend_radar_mask.py  1
- lib/improver/tests/acceptance/test_percentiles_to_probabilities.py  1
- lib/improver/tests/acceptance/test_nbhood.py  1
- lib/improver/tests/acceptance/test_nbhood_iterate_with_mask.py  3
- lib/improver/tests/acceptance/test_resolve_wind_components.py  1
- lib/improver/tests/acceptance/test_combine.py  1
- lib/improver/tests/acceptance/test_weighted_blending.py  2
- lib/improver/tests/acceptance/test_wind_downscaling.py  3
- lib/improver/tests/acceptance/test_threshold.py  1
- lib/improver/tests/acceptance/test_orographic_enhancement.py  3
- lib/improver/tests/acceptance/test_update_grid_metadata.py  2
- lib/improver/tests/acceptance/test_wind_direction.py  1
- lib/improver/tests/acceptance/test_wind_gust_diagnostic.py  1
- lib/improver/tests/acceptance/test_create_grid_with_halo.py  1
- lib/improver/tests/acceptance/acceptance.py  9
- lib/improver/tests/acceptance/test_generate_landmask_ancillary.py  1
- lib/improver/tests/acceptance/test_recursive_filter.py  1
- lib/improver/tests/acceptance/test_nbhood_land_and_sea.py  3
- lib/improver/tests/acceptance/test_temp_lapse_rate.py  2
- lib/improver/tests/acceptance/test_nowcast_optical_flow.py  3
- lib/improver/tests/acceptance/test_standardise.py  2
- lib/improver/tests/acceptance/test_uv_index.py  2
- lib/improver/tests/acceptance/test_percentile.py  1
- lib/improver/tests/acceptance/test_neighbour_finding.py  2
- lib/improver/tests/acceptance/test_wxcode.py  3
- lib/improver/tests/conftest.py  1
- lib/improver/tests/acceptance/test_snow_falling_level.py  2
- lib/improver/tests/acceptance/test_feels_like_temp.py  2
- lib/improver/tests/acceptance/test_wet_bulb_temperature.py  2
- lib/improver/tests/acceptance/test_probabilities_to_realizations.py  2
- lib/improver/tests/acceptance/test_percentiles_to_realizations.py  3
- lib/improver/tests/acceptance/test_spot_extract.py  3
- lib/improver/tests/acceptance/test_apply_emos_coefficients.py  2
- lib/improver/tests/acceptance/test_blend_adjacent_points.py  2
- lib/improver/tests/acceptance/test_estimate_emos_coefficients.py  2
- lib/improver/tests/acceptance/test_generate_topography_bands.py  2
- lib/improver/tests/acceptance/test_temporal_interpolate.py  5

Clones added
============
- lib/improver/tests/acceptance/test_nowcast_extrapolate.py  2
- lib/improver/tests/acceptance/test_nbhood_iterate_with_mask.py  1
- lib/improver/tests/acceptance/test_wind_downscaling.py  5
- lib/improver/tests/acceptance/test_orographic_enhancement.py  2
- lib/improver/tests/acceptance/test_wind_direction.py  1
- lib/improver/tests/acceptance/test_nowcast_optical_flow.py  2
- lib/improver/tests/acceptance/test_standardise.py  2
- lib/improver/tests/acceptance/test_percentile.py  2
- lib/improver/tests/acceptance/test_wxcode.py  1
- lib/improver/tests/acceptance/test_snow_falling_level.py  1
- lib/improver/tests/acceptance/test_apply_emos_coefficients.py  2
- lib/improver/tests/acceptance/test_estimate_emos_coefficients.py  2

See the complete overview on Codacy

@TomekTrzeciak @arh89 I've pushed new commits on this branch in response to the review comments and to update for PR #1019.


grad(jit(scan)) bug

import jax.numpy as np
from jax import grad, jit
from jax.lax import scan

f = lambda carry, _: ({k : v+carry['a'] for k,v in carry.items()}, np.zeros(()))

# this works
def loss(param):
  init_carry = {'a': param, 'b': np.zeros(())}
  final_carry, ys = scan(f, init_carry, np.array([1,2]))
  return final_carry['a'] * param
test = grad(loss)(np.zeros(()))

# this works
@jit
def loss(param):
  init_carry = {'a': param, 'b': np.zeros(())}
  final_carry, ys = scan(f, init_carry, np.array([1,2]))
  return final_carry['a'] * param
test = loss(np.zeros(()))

# this works
@jit
def loss(param):
  init_carry = {'a': param, 'b': np.zeros(())}
  final_carry, ys = scan(f, init_carry, np.array([1,2]))
  return final_carry['a'] * param + 0. * final_carry['b']
test = grad(loss)(np.zeros(()))

# this doesn't work
@jit
def loss(param):
  init_carry = {'a': param, 'b': np.zeros(())}
  final_carry, ys = scan(f, init_carry, np.array([1,2]))
  return final_carry['a'] * param
test = grad(loss)(np.zeros(()))

the last one gives TypeError: ((), ShapedArray(float32[]))

Looks like this was fixed, probably in #1224! Sorry for letting this issue slip through the cracks, but #1224 was essentially our attempt to categorically remove all remaining scan bugs at once.

@mattjj There might still be a bug somewhere. I've run into errors running this code:
import jax
import jax.numpy as np

def loop_stuff(init_x, func, num_iter):

    def body(args):
        i, x_new, _ = args
        x = init_x
        x_old = x

        x = func(i, x_old)
        i_new = i + 1
        return i_new, x_new, x_old

    init_vals = (0,
                 init_x,
                 np.zeros_like(init_x))

    def scan_step(args, idx):
        del idx
        return body(args), None

    (iterations, sol, prev_sol), _ = jax.lax.scan(
        f=scan_step,
        init=init_vals,
        xs=np.arange(num_iter),
    )
    return iterations, sol, prev_sol

max_steps = 10

def step(i, x):
    return x - 1

init_x = np.zeros(())

def run_scan(x):
    return loop_stuff(init_x=x, func=step, num_iter=max_steps)

@jax.jit
def jit_run_scan(x):
    return loop_stuff(init_x=x, func=step, num_iter=max_steps)

jax.grad(lambda x: run_scan(x)[1])(init_x)  # runs fine
jax.grad(lambda x: jit_run_scan(x)[1])(init_x)  # fails mysteriously

The error:
RuntimeError: Invalid argument: The parameter of condition and body, the result of the body, and init must all have the same shape; got Condition: (parameter: (s32[], (), (), (), (), (), f32[], (), ())) -> pred[]; body: (parameter: (s32[], (), (), (), (), (), f32[], (), ())) -> (s32[], (), (), (), (), (), f32[], f32[], ()); init: (s32[], (), (), (), (), (), f32[], (), ())..

Definitely looks like a bug! Can you open a new issue for that one?


[$250] LHN - Unread marker from second message when viewing conversation

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 9.0.14-1
Reproducible in staging?: Y
Reproducible in production?: N
If this was caught during regression testing, add the test name, ID and link from TestRail: https://expensify.testrail.io/index.php?/tests/view/4784111
Email or phone of affected tester (no customers<EMAIL_ADDRESS>Issue reported by: Applause - Internal Team
Action Performed:

Go to https://staging.new.expensify.com/
Navigate to the conversation with the secondary user. Be sure to clear any unread messages if needed
Keep the conversation in view of the most recent message
As the secondary user - Send some messages to the main user

Expected Result:
Verify the unread marker is not displayed in the conversation history over the new messages
Verify the conversation is not bold in LHN
Actual Result:
Conversation is displayed bold in LHN starting from second sent message
Unread marker is displayed in the conversation history over messages once away and back to conversation
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[ ] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
Add any screenshot/video evidence

https://github.com/user-attachments/assets/7e6fb0ef-d4d5-4b9e-ab04-ef91256a84f5
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01b903d59d2cb795a3
Upwork Job ID:<PHONE_NUMBER>138330554
Last Price Increase: 2024-07-30

Issue OwnerCurrent Issue Owner: @rayane-djouah

We think that this bug might be related to #vip-vsp

Just out of curiosity - why is the title in Russian ?

@hannojg sorry,  corrected

Proposal
Please re-state the problem that we are trying to solve in this issue.
Conversation is displayed bold in LHN starting from second sent message
Unread marker is displayed in the conversation history over messages once away and back to conversation
What is the root cause of that problem?
The condition here https://github.com/Expensify/App/blob/3f025eb2649b99b78d16322032f527f9a8033a68/src/pages/home/report/ReportActionsList.tsx#L467 is wrong in the case of scrollingVerticalOffset.current < MSG_VISIBLE_THRESHOLD. If that is true, we need to check reportAction.created > (userActiveSince.current ?? '') (or >=) instead of reportAction.created < (userActiveSince.current ?? ''), because the former means that the report action is created after the user was active on the report, thus should be shown, and isWithinVisibleThreshold and subsequently shouldDisplay should be true
What changes do you think we should make in order to solve the problem?

Update https://github.com/Expensify/App/blob/3f025eb2649b99b78d16322032f527f9a8033a68/src/pages/home/report/ReportActionsList.tsx#L467 to
const isWithinVisibleThreshold = scrollingVerticalOffset.current < MSG_VISIBLE_THRESHOLD ? reportAction.created > (userActiveSince.current ?? '') : true;

What alternative solutions did you explore? (Optional)

Thanks @ishpaul777. Can you put your PR up for review? And I'll merge it and get it CPd.

Hello, i read your job post for github issue.
that was fixed?

Contributor details
Your Expensify account email<EMAIL_ADDRESS>Upwork Profile Link:  https://www.upwork.com/freelancers/~01a5523c1adec76313

you have issues yet, please let me know.

Confirmed that the revert/CP fixed it. Removing blocker labels.

It looks like this is the PR: https://github.com/Expensify/App/pull/45240
It was raised by @ishpaul777 and reviewed by @mkhutornyi, so both are due payment.
Does that sound right?

Oh I see. Well glad I asked then 😅

Paid. Thanks @ishpaul777!

Paid. Thanks @ishpaul777!


iPad zoom issue

Current behavior:
iPad triggers zoom event on page scroll.
Expected behavior:
Do not trigger zoom event while scrolling the page
Details:
iOS: 12.4
Browsers: Chrome (74.0.3729.155) & Safari

Are you using d3-zoom 1.8.3?
I’ll also require further explanation because I don’t understand the description. The past release did not properly suppress scrolling when zooming on iOS, but I’ve never heard of zooming being triggered by scrolling.

I found a source of issue and it is comes from onresize event.
This issue can be closed.


Test: A trivial PR

What does this PR do?

Fixes # (issue)
Checklist

[ ] This PR fixes a minor issue (e.g.: typo or small bug) or improves the docs (you can dismiss the other
checks if that's the case).
[ ] This change is discussed in a Github issue/
discussion (please add a
link).
[ ] The documentation and docstrings adhere to the
documentation guidelines.
[ ] This change includes necessary high-coverage tests.
(No quality testing = no merge!)

Codecov Report

Merging #1836 (6586e35) into main (ef293be) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1836   +/-   ##
=======================================
  Coverage   70.32%   70.32%
=======================================
  Files          58       58
  Lines        4866     4866
=======================================
  Hits         3422     3422
  Misses       1444     1444

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ef293be...6586e35. Read the comment docs.


chore: Ping SaaS on login

Codecov Report

Merging #1424 (a786275) into master (9ffd4c6) will decrease coverage by 0.18%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1424      +/-   ##
==========================================
- Coverage   99.13%   98.95%   -0.19%
==========================================
  Files          84       86       +2
  Lines        6490     6514      +24
  Branches     1138     1110      -28
==========================================
+ Hits         6434     6446      +12
- Misses         29       36       +7
- Partials       27       32       +5

Flag
Coverage Δ

unittests
98.95% <100.00%> (-0.16%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

src/schemathesis/cli/__init__.py
99.11% <100.00%> (+0.01%)
:arrow_up:

src/schemathesis/service/__init__.py
100.00% <100.00%> (ø)

src/schemathesis/service/auth.py
100.00% <100.00%> (ø)

src/schemathesis/service/client.py
100.00% <100.00%> (ø)

src/schemathesis/service/constants.py
100.00% <100.00%> (ø)

src/schemathesis/service/metadata.py
100.00% <100.00%> (ø)

src/schemathesis/service/models.py
100.00% <100.00%> (ø)

src/schemathesis/specs/openapi/validation.py
76.47% <0.00%> (-11.77%)
:arrow_down:

src/schemathesis/_compat.py
70.37% <0.00%> (-7.41%)
:arrow_down:

src/schemathesis/runner/impl/threadpool.py
98.21% <0.00%> (-1.79%)
:arrow_down:

... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9ffd4c6...a786275. Read the comment docs.


FileNotFoundError: [Errno 2] No such file or directory: 'data/tets/256_tets.npz'

FileNotFoundError: [Errno 2] No such file or directory: 'data/tets/256_tets.npz'
how to get the file？Thanks~

Hi, I also meet the same problem. Can I ask how to solve this problem? (I find you close this issue and you may found the solution)
Thanks.

You may find the solution that in the GitHub repo of stabledreamfusion, there is a file of tets, you can get the answer in the generate.py
https://github.com/ashawkey/stable-dreamfusion/tree/main/tets
发件人: "yufeng9819" @.>
收件人: "guochengqian/Magic123" @.>
抄送: "ZYuhong" @.>, "State change" @.>
已发送邮件: 星期五, 8 月 4, 2023 07:28:53 PM
主题: Re: [guochengqian/Magic123] FileNotFoundError: [Errno 2] No such file or directory: 'data/tets/256_tets.npz' (Issue #10)
Hi, I also meet the same problem. Can I ask how to solve this problem? (I find you close this issue and you may found the solution)
Thanks.
—
Reply to this email directly, view it on GitHub , or unsubscribe .
You are receiving this because you modified the open/close state. Message ID: @.***>


[5.5] Add options parameter to paginate and simplePaginate

Currently on paginate and simplePaginate we're passing an array with with path and pathName, however Paginator and LengthAwarePaginator supports  path, query, fragment, pageName for the options arrray.
This PR adds the ability to pass these options to the Paginator...

@lucasmichot Changed, also changed the constructors for both Paginator and LengthAwarePaginator since they had the typehint too..

We have setters for those things. You can set them once you have the pagination instance.


How can we go back to previous questions?

I'm not part of the team but, there are at least two ways I know of to accomplish this: you can either click on the previous user input or use: window.ConversationalForm.remapTagsAndStartFrom function provided in wiki docs. Thanks!


added projects.json

made a file projects.json which took in all the projects that were in PROJECTS.md.
There is a name, title, description, link (GitHub), and a Twitter handle for each object.

Hey @AryanBabber thank you for this, can you explain what this project.json would be for and how we can keep  it up to date? thank you!

Yes, I'd love explaining it...
Basically, in order to add projects in project section, we need an object file, much like contributors.json where we listed all the names of the contributors.
Here, I have made the projects.md in a json form which can be used.

And for now, I can only think of manually updating it.

perfect explanation


Update to gradle 4.0 (#434)

(cherry picked from commit 7d91f470803115e988e59f4dd875309899101209)

Codecov Report

Merging #439 into v0.5.x will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             v0.5.x     #439   +/-   ##
=========================================
  Coverage     90.18%   90.18%
  Complexity      582      582
=========================================
  Files           100      100
  Lines          2058     2058
  Branches        209      209
=========================================
  Hits           1856     1856
  Misses          148      148
  Partials         54       54

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 924c23b...7cd1a37. Read the comment docs.


Remove publicKey from did document model

https://github.com/hyperledger/aries-framework-go/blob/baaee32af638b505f462db428bff5ef62b77458d/pkg/doc/did/doc.go#L317
publicKey was removed from the did-core abstract data model a while ago.

For now, we still need to support older DID methods that have the publicKey property.
I assume this is why rawDoc currently has the publicKey property.
@fqutishat @llorllale

@troyronda @fqutishat can you confirm which methods these are?

I assume technically did:peer is one of those.... :disappointed:


Fixes for Office 365 Outlook

Doesn't do unread email numbers. I've not figured that out yet.
But it's a start.
This is different to the built-in outlook.com. Microsoft keeps those separate for some reason.

Unread email numbers:
$('span[autoid=_n_V]').eq(0).text();
This gives me the unread count for my first favorite folder. Does it also work for you?
This would be a hacky solution until we find some better way to get it.

A hacky solution is better than no solution :)

https://github.com/meetfranz/plugins/commit/ed5a157d5159ed272f6d236e52c68af4d6992198
I guess we can close this now.

Yeah.
https://github.com/meetfranz/plugins/commit/ed5a157d5159ed272f6d236e52c68af4d6992198#diff-3127865a047c39095085be22dd60cf19
Looks better than my version :)

However, there's a typo in that one.
I'll update this PR to fix it.

Ah thank you! That had been annoying me for a while and hadn't got round to fixing it! :-)

Not half as much as having to use Office365 in the first place is annoying me ;)

The title of my inbox is unfortunately 'Inkorg' (Swedish) so the unread-mail counter does not work for me

Hello,
How is this office365 plugin?
Cannot pass login page after putting my company's email address. It says redirecting and then a blank page.
Thanks in advance.

Works fine for me. It's https://outlook.office365.com

maybe I'm doing something wrong.
I have to be redirected here: https://outlook.office.com/owa/?realm=nokia.com
But after login I just get a blank page.

Yes, that sounds correct.
Presumably you do not get the same issue in your browser?

no, I don't, tried with Firefox and Chrome ... Later I can try again from outside the company's network, just to test if  I run into the same issue.

Hello, I think that's some problem with the proxy.
I've tried to connect from home (same PC, no proxy) and I can use the plugin successfully.
Can you help me find the logs for Franz and for your plugin? If so, I can send some logs.
thx


Support cuda 7.5 and cudnn 7.0

Please, add support of cuda 7.5 and cudnn 7.0.
The best way would be to introduce it as configuration option. Right now, I have to correct everything manually, since only cuda 7.0 and cudnn 6.5 are supported

There are two short-term options for now.

Install Cuda 7.0 and Cudnn 6.5, in additional to your existing Cuda 7.5 and Cudnn 7.0 libraries. They are the officially supported versions. And it is okay to have multiple Cuda toolkits with different paths on your system. This is the recommended way.

Manually correct source code to pick up Cuda 7.5. We are discussing options to make it easier to configure in the future. However, given Cuda toolkits and libraries are not necessarily backward compatible, you may or may not hit subtle issues down the road.

I have Cuda 7.0 and Cudnn 7.0 installed on my machine. This message pop up when I import tensor flow:
I tensorflow/stream_executor/dso_loader.cc:77] LD_LIBRARY_PATH: :/usr/local/cuda-7.0/lib64
I tensorflow/stream_executor/cuda/cuda_dnn.cc:1062] Unable to load cuDNN DSO.
Does it mean it does NOT support Cudnn 7.0?

At this moment, the officially supported CUDNN version is 6.5 V2. This may change soon.
You can download the previous version from NVIDIA website, and use it in the configure script. This is okay even if you have CUDNN 7.0 V3 installed in the system.
https://developer.nvidia.com/rdp/cudnn-archive

Thanks! It helps. I got another problem when I tries to build from source
on another machine (Nvidia Dev Box, Ubuntu 14.04.1).
When I follow the Installation for linux from tensorflow offical site,
everything is fine until the last step:Train your first TensorFlow neural
net model.
When I execute the command, python
tensorflow/models/image/mnist/convolutional.py, An ImportError promps.
Traceback (most recent call last): File
"tensorflow/models/image/mnist/convolutional.py", line 13, in
import tensorflow.python.platform File "tensorflow/tensorflow/init.py",
line 4, in  from tensorflow.python import * File
"tensorflow/tensorflow/python/init.py", line 13, in  from
tensorflow.core.framework.graph_pb2 import * ImportError: No module named
core.framework.graph_pb2
Would you shed some light on this?
Thanks,
Shen
On Mon, Nov 9, 2015 at 11:27 PM, zheng-xq<EMAIL_ADDRESS>wrote:

At this moment, the officially supported CUDNN version is 6.5 V2. This may
change soon.
You can download the previous version from NVIDIA website, and use it in
the configure script. This is okay even if you have CUDNN 7.0 V3 installed
in the system.
https://developer.nvidia.com/rdp/cudnn-archive
—
Reply to this email directly or view it on GitHub
https://github.com/tensorflow/tensorflow/issues/54#issuecomment-155296419
.

De-duping with https://github.com/tensorflow/tensorflow/issues/20


Deployment failed

Hey, when trying to deploy the template in chapter 2 it fails. The installscript1 is showing a conflict status with the following error.
{"code":"VMExtensionProvisioningError","message":"VM has reported a failure when processing extension 'installscript1' (publisher 'Microsoft.Compute' and type 'CustomScriptExtension'). Error message: \"Command execution finished, but failed because it returned a non-zero exit code of: '1'. The command had an error output of: '.NET Framework 4.8 was installed, but a reboot is required. \r\n Please reboot the system and try to install/upgrade Chocolatey again.\r\nAt C:\\Windows\\TEMP\\chocolatey\\chocoInstall\\tools\\chocolateysetup.psm1:815 char:11\r\n+           throw \".NET Framework 4.8 w...' For more information, check the instance view by executing Get-AzVmssVm or Get-AzVm (https://aka.ms/GetAzVm). These commands can be executed using CloudShell (https://aka.ms/CloudShell)\". More information on troubleshooting is available at https://aka.ms/VMExtensionCSEWindowsTroubleshoot. "}
Any tips on resolving this ?

To get around this issue I installed .Net 4.8 on the VM and ran the installscript1 within the VM.
If possible it would be good to adjust the template so the script installs without this issue.
Leaving this link as refrence if anyone else runs into issues with this :)
https://github.com/microsoft/AL-Go/issues/560


Update sql-data-warehouse-develop-connections.md

I think it's more clear this way, since I didn't know what MPP is, so I had to google it.

Hi @4c74356b41, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!

This seems like a small (but important) contribution, so no contribution license agreement is required at this point. Real humans will now evaluate your PR.

TTYL, AZPRBOT;

@4c74356b41
I'm convinced! Thank you for your contribution.
--Carolyn

@barbkess


Streamline observations download

The aim of this pull request is to streamline the procedure to download and reformat observational datasets for ESMValTool. I do not aim to have automated downloads for every dataset we are currently supporting, but to provide the framework and a good number of examples of implementation so it becomes easy to implement in the near future.
In particular, I have created three basic downloader classes for HTPPS (using wget), FTP (using the python library) and Climate Data Store (using cdsapi) downloads along with some example implementations for some datasets.  I also provide some derived classes to simplify getting dataset from specific sources, like the NASA storage or the ESA-CCI CEDA anonymous FTP.
Regarding the interface, I plan to use the reworked CLI interface in ESMValGroup/ESMValCore#605 to refactor it so we get something similar to:
esmvaltool data prepare DATASET1,DATASET2 --start 1990 --end 2015
esmvaltool data download DATASET1,DATASET2 --start 1990 --end 2015
esmvaltool data format DATASET1,DATASET2 --start 1990 --end 2015
where prepare will download and format the data in one go. Hopefully, this will make life easier for those user that do not have access to JASMIN nor DKRZ.
Tasks

[x] Create an issue to discuss what you are going to do, if you haven't done so already (and add the link at the bottom)
[x] Give this pull request a descriptive title that can be used as a one line summary in a changelog
[x] Add documentation
[x] Make sure your code is composed of functions of no more than 50 lines and uses meaningful names for variables
[x] Circle/CI tests pass. Status can be seen below your pull request. If the tests are failing, click the link to find out why.
[ ] Preferably Codacy code quality checks pass, however a few remaining hard to solve Codacy issues are still acceptable. Status can be seen below your pull request. If there is an error, click the link to find out why. If you suspect Codacy may be wrong, please ask by commenting.
[ ] Please use yamllint to check that your YAML files do not contain mistakes
[x] (Only if really necessary) Add any additional dependencies needed for the diagnostic script to setup.py, esmvaltool/install/R/r_requirements.txt or esmvaltool/install/Julia/julia_requirements.txt (depending on the language of your script) and also to package/meta.yaml for conda dependencies (includes Python and others, but not R/Julia)
[x] If new dependencies are introduced, check that the license is compatible with Apache2.0

Cool idea @jvegasbsc! Maybe we could choose a different name than obs or observations, because not everything we cmorize are observations? See also #1407.
I'm not too sure about inclusion as a subcommand in the esmvaltool command, because that is part of ESMValCore while the CMORizers live in ESMValTool.

I'm not too sure about inclusion as a subcommand in the esmvaltool command, because that is part of ESMValCore while the CMORizers live in ESMValTool.

I think the discoverability we will get with a single entry point is worth it. But my idea will be to find a way to allow ESMValTool to add commands to the interface without ESMValCore knowing anything about them, like a plug-in or similar

@jvegasbsc this is very cool! But I have an important question (maybe @mattiarighi can also shed more light on it too): what are the licensing troubles we may get into if some random user downloads Tier 3 data via ESMValTool and starts redistributing it against the rules?

what are the licensing troubles we may get into if some random user downloads Tier 3 data via ESMValTool and starts redistributing it against the rules?

In that case, the illegal part is to redistribute, not to download. We are not supporting that part, just the downloading procedure, which is completely fine

what are the licensing troubles we may get into if some random user downloads Tier 3 data via ESMValTool and starts redistributing it against the rules?

In that case, the illegal part is to redistribute, not to download. We are not supporting that part, just the downloading procedure, which is completely fine

there is a fine line between download and redistribution - we should make sure that the tool doesn't write any freshly downloaded Tier 3 files to any location before CMORizing them but rather downloads and CMORizes them on the fly; there is also the issue of clogging the disk if the tool keeps the originally downloaded files :beer:

I'm not too sure about inclusion as a subcommand in the esmvaltool command, because that is part of ESMValCore while the CMORizers live in ESMValTool.

I found a way to allow adding subcommands in esmvaltool without ESMValCore knowing anything about it. It is one of the ways that pytest uses to discover plugins.
Step 1: define a esmvaltool_commands entry point in setup.py:
'esmvaltool_commands': [
    'data = esmvaltool.cmorizers.obs.cmorize_obs:DataCommand'
]

In ESMValCore, scan for 'esmvaltool_commands' entry points and add the classes to the Fire interface:
for ep in iter_entry_points('esmvaltool_commands'):
    if hasattr(self, ep.name):
        logger.error('Registered command %s already exists', ep.name)
        continue
    self.__setattr__(ep.name, ep.load())

so we can decouple completely ESMValCore from this.

Would it be an idea to add a command like
esmvaltool download recipe_example.yml

that would download and reformat all data needed to run recipe_example.yml? That way users would not need to bother with figuring out what data is needed exactly to run a recipe.
Maybe that could be simplified even further to
esmvaltool run --download recipe_example.yml

to combine all steps of obtaining the data, formatting it and running into a single command.

This is now ready to test. The main functionalities to test are:
esmvaltool data list: to show all datasets supported by ESMValTool cmorizers
esmvaltool data info DATASET: to show detailed info about DATASET.
I would like to have feedback in the info that we should show in both function and also in the format
esmvaltool data prepare DATASET: to automatically download and format. Please, test also the flags available
esmvaltool data download DATASET
esmvaltool data format DATASET
Those two are to download and format in two steps. There is no need to test them separated from the prepare for each dataset.

I tried to use this branch to download and cmorize HadCRUT4 data. Here's my feedback:
esmvaltool data list: works, nice!
esmvaltool data info HadCRUT4: doesn't work (cannot find key). How should it be formatted?
esmvaltool data prepare HadCRUT4: complains about my config rootpath. How can I add a custom config file with this command? After modifying the default config file it downloads the files as expected. Nice! However, it gets stuck in cmorizing, NCL fails with a fatal message, but this error is not returned to the main program, which therefore hangs indefinitely.

esmvaltool data info HadCRUT4: doesn't work (cannot find key). How should it be formatted?

Fixed. There was a clash of names between the operator and a class attribute. Your call should work now

You can use the flag --config_file to specify a custom config user

esmvaltool data prepare HadCRUT4: complains about my config rootpath. How can I add a custom config file with this command? After modifying the default config file it downloads the files as expected. Nice! However, it gets stuck in cmorizing, NCL fails with a fatal message, but this error is not returned to the main program, which therefore hangs indefinitely.

I forgot to update some paths when I renamed the obs folder. It should work now. @pcosbsc

The tests are working for me locally. Can any of you please test it locally to see if it is something from my enviroment?

The problem with ci/circleci test is that you're trying to read files from the source (which is removed to make sure we're testing the installation) rather than from the installed copy.

Could you move everything one level up, so move everything from the esmvaltool/cmorizers/data directory to esmvaltool/cmorizers? I think we don't need the extra subdirectory data.

The test from ci/circleci: develop also happens when I run the tests locally, maybe you have a different version of datasets.yml or recipe_check_obs.yml that you forgot to check in?
    def test_datasets_are_added_to_test_recipe():
        with open(os.path.join(yaml_folder, 'datasets.yml'), 'r') as file:
            cfg = yaml.safe_load(file)

        recipe_path = os.path.join(recipes_folder, 'examples/recipe_check_obs.yml')
        with open(recipe_path, 'r') as file:
            recipe = yaml.safe_load(file)

        tested_datasets = set()
        for diagnostic in recipe.get('diagnostics', {}).values():
            for dataset in diagnostic.get('additional_datasets', {}):
                tested_datasets.add(dataset['dataset'])
            for variable in diagnostic.get('variables', {}).values():
                if variable is None:
                    continue
                for dataset in variable.get('additional_datasets', {}):
                    tested_datasets.add(dataset['dataset'])

        for dataset in cfg['datasets'].keys():
>           assert dataset in tested_datasets
E           AssertionError: assert 'ESACCI-SEASURFACE-SALINITY' in {'APHRO-MA', 'AURA-TES', 'BerkeleyEarth', 'CALIPSO-GOCCP', 'CDS-SATELLITE-ALBEDO', 'CDS-SATELLITE-LAI-FAPAR', ...}

Sorry I promised I would review but haven't had the time. Would it make sense to start reviewing now and see if this PR can be included in the upcoming release?

Thanks for the great work already @jvegasbsc! I have started the review of this PR and will try to review bits of code regularly. Perhaps we would need a backwards incompatible change label for this PR.

Several cmorizers have been merged since this PR was set to RfR or are currently being developed. I'm starting a table here to keep track of these cmorizers. Feel free to edit.

Dataset
Tier
PR
Comments

ESACCI-OZONE
2
#1832
PR for the cmorizer not approved. It would be better to have it approved because scientific reviews cannot be redone here

GCP
2
#2295
Cmorizer will be updated. Not yet approved.

Hi @jvegasbsc! Just to let you know that I have reviewed half of the files, including all downloader scripts. This is nice work already! Most of the downloaders work like a charm 👍  I have put my comments whenever there is a minor issue with a downloader script. Most of the time it is about missing default start_date and end_date or wrong tier because tier=3 is the default. I note that none of the Tier3 datasets could be downloaded without storing my credentials in a .netrc file and sometimes signing licences for which the links are displayed in the logs. We may just need to add some documentations about the download process of Tier3 datasets. I will continue with the cmorizer scripts asap.

Thanks for all the feedback. I will start integrating your comments and fixing the ones that do not work

Ok, all issues with the cmorization interface should be fixed now, as I added a test checking that all have the required number of arguments, and even their names.

Hi @jvegasbsc, I have just reviewed all downloader scripts and almost all work fine. Thanks for the changes! There are a handful that would need a bit more attention. I'm getting a certificate issue to download a few datasets (GISTEMP, ISCCP-FH, NSIDC-0116-*h), see below. Would you know what to do in such cases? Should one use --no-check-certificate as an optional argument and how?
2021-12-06 10:57:38,639 UTC [12085] INFO    Downloading original data...
2021-12-06 10:57:38,647 UTC [12085] INFO    Downloading GISTEMP
2021-12-06 10:57:38,647 UTC [12085] DEBUG   Download module: gistemp
2021-12-06 10:57:38,647 UTC [12085] DEBUG   ['wget', '--no-clobber', '--directory-prefix=/scratch/b/b309192/data/obs_PR/RAWOBS/Tier2/GISTEMP', '--no-directories', 'https://data.giss.nasa.gov/pub/gistemp/gistemp250_GHCNv4.nc.gz']
--2021-12-06 11:57:38--  https://data.giss.nasa.gov/pub/gistemp/gistemp250_GHCNv4.nc.gz
Resolving data.giss.nasa.gov... <IP_ADDRESS>, 2001:4d0:2310:230::233
Connecting to data.giss.nasa.gov|<IP_ADDRESS>|:443... connected.
ERROR: cannot verify data.giss.nasa.gov's certificate, issued by `/C=US/O=Let\'s Encrypt/CN=R3':
  Issued certificate has expired.
To connect to data.giss.nasa.gov insecurely, use `--no-check-certificate'.
2021-12-06 10:57:39,069 UTC [12085] ERROR   Failed to download GISTEMP
Traceback (most recent call last):
  File "/mnt/lustre01/pf/b/b309192/obs_PR/ESMValTool/esmvaltool/cmorizers/data/cmorizer.py", line 127, in download
    self.download_dataset(dataset, start_date, end_date, overwrite)
  File "/mnt/lustre01/pf/b/b309192/obs_PR/ESMValTool/esmvaltool/cmorizers/data/cmorizer.py", line 164, in download_dataset
    downloader.download_dataset(self.config, dataset,
  File "/mnt/lustre01/pf/b/b309192/obs_PR/ESMValTool/esmvaltool/cmorizers/data/downloaders/datasets/gistemp.py", line 34, in download_dataset
    downloader.download_file(
  File "/mnt/lustre01/pf/b/b309192/obs_PR/ESMValTool/esmvaltool/cmorizers/data/downloaders/wget.py", line 55, in download_file
    subprocess.check_output(command)
  File "/work/bd0854/b309192/soft/miniconda3/envs/tool_obs_24/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/work/bd0854/b309192/soft/miniconda3/envs/tool_obs_24/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['wget', '--no-clobber', '--directory-prefix=/scratch/b/b309192/data/obs_PR/RAWOBS/Tier2/GISTEMP', '--no-directories', 'https://data.giss.nasa.gov/pub/gistemp/gistemp250_GHCNv4.nc.gz']' returned non-zero exit status 5.
2021-12-06 10:57:39,076 UTC [12085] ERROR   Download failed for datasets ['GISTEMP']

I have also retried many formatter scripts and will start checking the output soon.

The NASA downloads are working for me and the certificate looks well when entering through the browser

The valid period is just two months, so maybe it was expired you tried it and they just updated it after that?

I will rework the file naming of the CDS downloaders, as it is not really clear or informative

The NASA downloads are working for me and the certificate looks well when entering through the browser
The valid period is just two months, so maybe it was expired you tried it and they just updated it after that?

Yes I see the same when checking that certificate but I've not managed to use wget for these 4 datasets without the --no-check-certificate option. After adding it temporary to wget.py, I could test these 4 downloaders successfully. But it seems to me that the issue in on the server side.

Finally, we need a plan to proceed to the merge. Shall we announce this soon (now?) on GitHub because this will affect cmorizers currently under development? I'm not sure if it's better to merge this PR first and help developers to adjust to the new framework or to help developers to merge their contribution quickly and we later include those to this PR. What do you think?

We should merge and help. There will be no time in which new cmorizers are not under development to do the merge without disruption. It is also better if we do it now for testing purposes, as this is a big change and there is enough time until the next release to maybe get a good chunk of testing from other partners

We should merge and help. There will be no time in which new cmorizers are not under development to do the merge without disruption. It is also better if we do it now for testing purposes, as this is a big change and there is enough time until the next release to maybe get a good chunk of testing from other partners

Sounds good to me. I forgot to write that I was done updating the instructions. This includes explaining the new cmorizer framework and fixing all the paths and filenames. Please carry on with the review of the *rst files.
Besides, I will make a quick PR to adjust the Core documentation to the new framework.

@ESMValGroup/esmvaltool-coreteam since this is quite a big change, some of you may want to have a final look at this PR before it is merged. As planned with my co-release managers @axel-lauer and @schlunma, we are still aiming at merging that before the release, maybe by mid next week. As soon as it is merged, a GitHub discussion will be open to inform the community that a new cmorization framework is available for testing with instructions on how to proceed. Hope that is agreeable.

It seems to me that the Codacy check was somehow disabled before because previous commits were all green and the 274 (!) issues are not (only) related to the last 3 commits... And I'm not sure what to do with these 274 Codacy issues...

Thanks Rémi for driving this forward! My view on the issues you mentioned is:

"Too many {arguments|local variable}": would be good to avoid, but can be ignored, particularly since it is mostly not outrageous (something like (6/5) or similar).
Formatting that could be an f-string: Usually very easy thing to improve. Perhaps something we can still do?
Similar lines in 2 files: In my experience mostly bogus. Ignore.
Locally disabling: Ignore
FTP: Well, not much we can do if that's how the data is provided. Ignore.

Here's my take on the Codacy issues:

We also have 44: Argument name "__" doesn't conform to '[a-z_][a-z0-9_]{2,30}$' pattern ('[a-z_][a-z0-9_]{2,30}$' pattern) which come from the new cmorizer functions, e.g.:
def cmorization(in_dir, out_dir, cfg, _, __, ___):

Any opinion on how to handle that?

Maybe try something like
def cmorization(in_dir, out_dir, cfg, *_):

?
About locally disabling pylint: I'm not in favour of doing this. There is also no need to do this. Let's remove the lines that locally disable certain warnings so we can see the real issues and actually monitor code quality.

Thanks a lot for your input @zklaus and @bouweandela! I looked at the use of pylint disabling statement more carefully and it seems that most, if not all, are related to these extra arguments in the 2 main functions of the new framework cmorization and download_dataset:
In def cmorization(in_dir, out_dir, cfg, _, __, ___), the 3 optional arguments are related to a custom config-user used for some cmorizers, a start date and an end date. I guess it was implemented like this to help developers of cmorizers understand which and how many optional arguments can be considered when developing cmorizers. Using *_ would certainly help to reduce codacy issues and getting rid of pylint: disable=invalid-name but would be harder to understand for developers.
We have the same situation in download_dataset(config, dataset, dataset_info, _, __, overwrite) where the optional arguments are a start and an end date. Since not all datasets are provided in yearly files or so, this implementation helps to homogenize the call to the download function between all datasets.
I think we need to decide between better code quality (less Codacy issues, no pylint: disable) and more clarity for developers of cmorizers. My suggestion would be to leave these 2 functions as is and maybe remove pylint: disable everywhere or add in all cases (currently not done). We could revise this later on after a few developers have started using this new framework in v2.5. Perhaps something to decide at the next monthly meeting?

I implemented the suggested change. However, this is of course still up for discussion. I'm happy (enough) to roll it back.

Thanks for your suggestion and implementation! I'm fine with your approach which clearly uses all possible arguments in the download and cmorizeration functions. I tested it on a few cases and everything seems to continue to work fine. The number of Codacy issues has increased but this is due to the many new "unused arguments" introduced. Not sure if we need pylint: disable =unused-argument statements.
I'll finish what I can with the formatting early next week and then I guess we can finally merge this PR.

@remi-kazeroni, I have parried down the number of codacy complaints to 17, mostly by fiddling with the config, but also by addressing some of the simpler ones. Could you have a go at the remaining ones and then we can resolve what remains together?

@remi-kazeroni, I have parried down the number of codacy complaints to 17, mostly by fiddling with the config, but also by addressing some of the simpler ones. Could you have a go at the remaining ones and then we can resolve what remains together?

Thanks a lot for your help and all your work Klaus! That is really great to have only a manageable number of issues to address. I will look into those asap.
I think this PR is a good practical opportunity to adjust the settings of the linting and formatting tools used in the PRs. Perhaps that is something worth mentioning at the next Tech Lead meeting and see if everyone agrees since these changes may be a bit hidden by the rest of the PR.
I have just finished retesting all cmorizers and all downloaders and apart from one case (Scripps-CO2-KUM), everything runs fine. In my previous review I ran downloaders and cmorizers separately. But I have realized that there were still a few bugs when using prepare (download+cmorize in one go) on some datasets. In particular, unpacking tarballs and deleting those in now part of the downloaders and this was not always removed from cmorizers. As a consequence, the RAWOBS folder happened to be emptied for some datasets after running the cmorizer. This is now fixed.

Great work, @remi-kazeroni. You are also right about the opportunity for fine-tuning. In fact, I have already prepared a branch locally with all the changed settings that are in this PR for now. I will open a new PR to discuss them before the tech lead meeting on Friday.

The codacy check appears green now, which is a good step forward, but really due to the arbitrary number of codacy points that we have chosen to accept (10). There are still a few that I think should definitely be addressed. Particularly the following:
* `Attribute 'config' defined outside __init__`

* `Either all return statements in a function should return an expression, or none of them should.`

Yes, I wanted to comment on that but I forgot. I think the Either all return statements in a function should return an expression, or none of them should. are not so easy to fix. I see the purpose of the implentation in cmorizer.py and why it triggers these issues. I have tried to get rid of these issues but there is always a use case that fails (e.g. using a wrong dataset name, wrong dates, a list of datasets)... The 2 faulty cmorizers with the same Codacy issue would require a bit of work. The original implementation is different from other cmorizers but this PR has nothing to do with that.
Btw, if we were to clean up every file of this PR, we could also have a look at yamllint. There are many files with too long lines, long urls that do not conform with yamllint. Would it be something to try to fix as well?

Btw, if we were to clean up every file of this PR, we could also have a look at yamllint. There are many files with too long lines, long urls that do not conform with yamllint. Would it be something to try to fix as well?

To quote myself on this:

When the tool complains about the maximum line length or too many spaces, please use your own best judgement about whether solving the issue will make your recipe more readable.

from the contribution guidelines. So I would not break up long URLs, that does not increase readability.

The codacy check appears green now, which is a good step forward, but really due to the arbitrary number of codacy points that we have chosen to accept (10). There are still a few that I think should definitely be addressed.

Yes, that is why the limit was set to 0. It's fine to merge with some issues, but they should preferably be discussed in the pull request.

Hi @zklaus, I have finished fixing the last (problematic) Codacy issue after deciding on the behaviour of the prepare command. I checked that the latest changes do not break anything w.r.t. the cmorizers. Do you think you would have a chance to look at the last changes over the next couple of days and approve if fine for you? Could you please also let me know if I can revert the changes regarding the configuration files for the linting tools?
We would still like to merge this PR this week, say Thursday, so that it lives in the main branch for a few days before the v2.5 release and we can see how the automatic testing engines react to that.

@remi-kazeroni, nice work! I'll have very few minor comments, but regarding the reverting of the linter configs: Absolutely, do revert them. In this branch, they were only intended to help filter the barrage of codacy problems so that we could see what really needs addressing.

Stellar work, @remi-kazeroni !

Thanks a lot @zklaus! All credits should go to @jvegreg for doing all the coding! 🍻
But I still wanted to address your final comment on the timerange selection. I agree the tool should return an error if the time range selection is wrong (wrong argument in the cli). The current assumption that a wrong timerange leads to the downloading/formatting of the whole datasets is not ideal in my opinion.
Please nobody merge now 😆

I have done some further tests on possible misuse of the time ranges arguments (--start, --end). For example, what happens with --start=1250? The download does not start but the error message is not uniform across datasets. This depends on the data source, the file naming (year included in the file name, folder name,...). Or what happens if a timerange is specified for a dataset where this is not needed (e.g. just one global file and no yearly files)? In these cases, the global files are downloaded.
In short: there may be room for further enhancements but I would leave that for future PRs since I have not found any problematic cases with the current handling of the time range.

Not sure how to display the codacy report of the second to last commit, but the number of issue was 7 without any problematic issue left. The number grew to 310 after reverting the changes on .pylintrc and .prospector.yml made by @zklaus to help with the handling of Codacy issues.
To be able to merge this, I guess we just need a final approval from @zklaus and coordinate the merge button hitting with @schlunma (I'd need a bit of time to draft a message to open a Github discussion about the changes related to the new cmorizer interface).
Thanks to all who contributed, reviewed, tested!! 🥳 🍻

Sounds great, @remi-kazeroni :+1: I will merge this PR shortly after you opened the discussion.
Thanks to everyone who contributed to this, this is an awesome addition :tada:

Not sure how to display the codacy report of the second to last commit, [...]

I'm not sure it's possible with Codacy. In principle, you can click on the little icon in front of the commit hash in the listing above (a green check mark, a red x, or a yellow dot for successful, failed, and ongoing checks, respectively). This will pop up a little dialog where you can click on details to see the specifics of a particular check. Have a look at the following screenshot.

Great, thanks to everyone, particularly @remi-kazeroni for pushing this over the finish line! :tada:
There certainly is room for improvement, but as you said, let's take that up later.

Thanks! @schlunma: could you please merge this now? It would help me to get the right links in the documentation to write the discussion.

@remi-kazeroni and the Streamliners band, legends! :beer:


Tx on Air Band

Hello! Thanks a lot for you wonderfull work. I have an aviation license so I can broadcast on aviation bands. What are the parameters that need to be modified (in the code) to be able to speak on those frequencies? Obviously at my own risk

What modulation ?
Remender that this radio only does FM wih TX.

Thx for aswer.
On AM.
With this firmware (https://whosmatt.github.io/uvmod/) I could transmit.

Only RX, no TX.

IMHO, don't use this radio on airband despite you have a licence to do.

Le ven. 24 nov. 2023 à 14:02, Giagio89 @.***> a écrit :

Hello! Thanks a lot for you wonderfull work. I have an aviation license so
I can broadcast on aviation bands. What are the parameters that need to be
modified to be able to speak on those frequencies? Obviously at my own risk
—
Reply to this email directly, view it on GitHub
https://github.com/egzumer/uv-k5-firmware-custom/issues/142, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ANIGEUJBAA56CDITSQGDWATYGCLFBAVCNFSM6AAAAAA7ZDHZ5CVHI2DSMVQWIX3LMV43ASLTON2WKOZSGAYDSNRZGUYDEMI
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Suggest to read this.... https://en.wikipedia.org/wiki/Airband

Harmonics are not a problem on those frequencies. The lack of AM modulation is. If you want to transmit on airband buy a proper airband radio. Airband communication is a serious thing, I'm not going to help you to spam on that band. Sorry.


Remove an unused meta file

The folder was deleted but the .meta file was not cleaned up.

/azp run


[cloud_firestore] isEqualTo matches against null

Original Issue
YouTrack
Can replicate
return StreamBuilder<QuerySnapshot>(
        stream: users.where("uid", isEqualTo: null).snapshots(),

returns all documents in collection, even if field uid is set to null in at least one document.

Closing: https://github.com/FirebaseExtended/flutterfire/issues/691#issuecomment-641786240
This would make sense to keep as is looking at the code, since all of the other filter types don't accept null, allowing isEqual to accept it would be confusing & potentially cause unwanted behaviour.


Issue with /docs/hellonode/ - error: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

The first command that uses kubectl results in this error:
error: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

Possibly relevant for reproduction: I initially selected us-west-1 for the first run and then switched to us-central-1 for the run documenting repro steps. I am currently in Portugal.
Full script copied below. Seems like it could be related to https://github.com/kubernetes/kubernetes/issues/28612
session.log.txt

I was able to resolve this by doing the following before any kubectl commands:
$ gcloud auth application-default login

can this be added to the hello world documentation? or was there a step I missed that should have done this?

The /docs/hello-node page has been replace with a page that doesn't use Google Cloud Platform:
https://kubernetes.io/docs/tutorials/stateless-application/hello-minikube/


fatal error: pcl/io/openni2_grabber.h: No such file or directory

When I clone and build with command

luarocks make rocks/torch-pcl-scm-1.rockspec

it output fatal error:

fatal error: pcl/io/openni2_grabber.h: No such file or directory
compilation terminated.
CMakeFiles/pcl.dir/build.make:206: recipe for target 'CMakeFiles/pcl.dir/src/xyzrgba.cpp.o' failed
make[2]: *** [CMakeFiles/pcl.dir/src/xyzrgba.cpp.o] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/pcl.dir/all' failed
make[1]: *** [CMakeFiles/pcl.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

Does anyone see this issue? Currently my setting: Ubuntu 16.04, pcl 1.8

Oh, my fault, my install pcl is lacking of openni2. I solved by

sudo apt-get install libopenni2-dev

then build and install pcl lib again.


Menu items have mouseover/highlighting display error

Screenshot

Description
Menu items in superset show a faint blue bar that's very tight to the text. The intent was to have the full item "bar" be that color, from left to right, with a little padding above/below.
Design input
[describe any input/collaboration you'd like from designers, and
tag accordingly. For design review, add the
label design:review. If this includes a design proposal,
include the label design:suggest]

Hi rusackas
I am interested in solving this issue, this is my first OSP, can you please explain how to solve this issue. Thank you


matrix-appservice-discord: 3.1.0 -> 3.1.1

Changelog: https://github.com/matrix-org/matrix-appservice-discord/blob/develop/CHANGELOG.md#311-2022-11-10

Quoting Winter (2022-11-21 14:45:30)

is there a reason you have it marked as only supporting Linux? Looks
like that may be a bit too restrictive, since it's just a Node
application. (If there's no Linux-specific code, which I'm 99.9% sure
about, I think this would be better set to nodejs.meta.platforms.)

The package has some native dependencies such as sqlite and now the Rust
Matrix cryptographic library. I'm not sure whether those build on other
platforms. Opening the build for more platforms seems reasonable, but I
would not be able to test those myself.


Opening a webpage with WebviewWindow encounters an error STATUS_ACCESS_VIOLATION.

Hi everyone,
I encountered a little issue while using Tauri. I wanted to ask whether this is a bug or something specific to my setup.
After the release of Tauri v2.0.0-beta.14, I eagerly upgraded my program from beta.8 because beta.14 fixed the issue of webviews not being created. However, just when everything seemed to be going well, I found that although my program could create webviews normally, the web page would crash with the error code STATUS_ACCESS_VIOLATION.This didn't occur in beta.8.

This issue is almost nonexistent on text web pages, but it occurs almost 100% of the time on video streaming platforms like YouTube and Twitch. Additionally, on websites with Cloudflare verification, it leads to verification failure.

After testing, I found that the issue occurs with.plugin(tauri_plugin_notification::init()). Removing this line resolves all the problems.
Since it seems to be related to tauri_plugin_notification, I reported it here. Can anyone help?
Thank you very much!!

Encountering this as well on production builds, exclusively. Dev seems to work fine.

This issue is stell present in tauri 2.0.0-beta.23

Did somebody try to repro this in a more minimal environent like a new CTA app? I'm having issues reproducing this so i assume there's a condition here i'm not seeing.

@FabianLars This is how I reproduce the issue. For your reference.
PS D:\Visual_studio\Tauri>  pnpm create tauri-app --beta
../../.pnpm-store/v3/tmp/dlx-17380       |   +2 +
../../.pnpm-store/v3/tmp/dlx-17380       | Progress: resolved 12, reused 2, downloaded 0, added 2, done
✔ Project name · tauri-app
✔ Choose which language to use for your frontend · TypeScript / JavaScript - (pnpm, yarn, npm, bun)
✔ Choose your package manager · pnpm
✔ Choose your UI template · React - (https://react.dev/)
✔ Choose your UI flavor · JavaScript
✔ Would you like to setup the project for mobile as well? · no
 cd tauri-app
 pnpm install
add tauri-plugin-notification = "2.0.0-beta"  to Cargo.toml
add  "webview:allow-create-webview-window" to capabilities/default.json
add .plugin(tauri_plugin_notification::init()) to main.rs
add
import { WebviewWindow } from '@tauri-apps/api/webviewWindow'
<button onClick={() => new WebviewWindow('test-1', { url: 'test url '})}>Open</button>  to App.jsx
For the test url, you can use YouTube or Twitch URLs.
I tested on tauri-beta.23, and the same issue I described still occurs.

I pulled it locally for testing and added some console.logs - i can't seem to break using debugger because the rollup process removes it and the window launches too fast before devtools.
One thing to note - we are connecting to a remote external URL when doing this. That is the only way this crashes. When we launch localhost, it runs fine.
(unknown) isPermissionGranted() default
(unknown) isPermissionGranted() entry calling plugin:notification|is_permission_granted
(unknown) true
(unknown) void isPermissionGranted() response true
(unknown) void isPermissionGranted() response is not null
(unknown) setNotificationPermission() granted
(unknown) setNotificationPermission() window.notification.permission set
(unknown) setNotificationPermission() permissionSettable set to false
index-C2VP2neN.js:703 unregistering
index-C2VP2neN.js:703 registering
index-C2VP2neN.js:703 requesting machine info
index-C2VP2neN.js:703 starting monitoring loop
// redacted
(unknown) isPermissionGranted() default
(unknown) isPermissionGranted() entry calling plugin:notification|is_permission_granted
(unknown) isPermissionGranted() default
(unknown) isPermissionGranted() entry calling plugin:notification|is_permission_granted

So here is where we get even stranger - if i set a breakpoint on is_permission_granted in the application, i only ever see it fire once.
If I set a breakpoint and just let the backend hold it there, the frontend still crashes with the same state and the log file still shows two additional calls happen before the window implodes.

If I replace this function with a simple Promise.resolve(true) - the app will not crash.

Thanks for the repro!! I'm quite busy over the weekend so i don't know when exactly i'll be able to look into it (it's high on my prio list tho)

Any progress on fixing this bug?

Nope, like i said i was unavailable over the weekend, and mondays are always busy as hell. I'll look into it tomorrow morning.
That said, i can at least confirm that i see the same issue using your repro, so thanks for that.

So i can't really tell what's really causing this, maybe chromium doesn't like that we call fetch inside the permission request, at least on "normal" origins (instead of our custom origin if you're not loading remote websites).
Currently i'd just lean towards having an OS check in the init js script and iterate later when we use webview2's permission apis. The Rust end always returns Granted anyways 🤷

@WalrusSoup

If I replace this function with a simple Promise.resolve(true) - the app will not crash.

Did notifications work afterwards?

@WalrusSoup

If I replace this function with a simple Promise.resolve(true) - the app will not crash.

Did notifications work afterwards?

No issue on this end when just returning true.
I think lazily evaluating the permissions when the actual function to show a notification is called may resolve it too. I have no idea why it fires two additional times when loading.
https://github.com/user-attachments/assets/bb6fa070-34ce-4c49-99e9-54a5b03501f2

I think lazily evaluating the permissions when the actual function to show a notification is called may resolve it too. I have no idea why it fires two additional times when loading.

Ahhh, i missed the last part of the init script where we check the permission state. I assume it's there for a reason but i'll look into that tomorrow.
No idea why it fires multiple times either. Are you loading iframes maybe?

No idea why it fires multiple times either. Are you loading iframes maybe?

IFRAMES!
Our app has Sentry performance monitoring with playback support. If i break on the second\third iteration of this, I do see that higher on the stack from the additional function calls does appear to be from Sentry.

It looks like the actual problem zone is located here which decides if the environment supports fetch natively by injecting an iframe into the header.


[WIP] - [Metrics ServerApp] - New metrics serverapp

PR Details

Description
PR migrates current ASP.NET project to ASP.NET Core + EntityFramework/Dapper. I've also rearranged code refactored code to make it more readable. Additionally, project has Swagger support to easily call endpoints.
Types of changes

[x] Docs change / refactoring / dependency upgrade
[x] Bug fix (non-breaking change which fixes an issue)

Todo

[ ] - Finish following endpoints :

[ ] ActivityMetrics.Controller. GetCrashesPerVersion - takes about 2 minutes to load. I was not able to materialize the query. Probably need to create a MetricCache table, create a job using i.e HangFire lib or SQL agent
[ ] ActivityMetrics.Controller.GetHighUsage - I need to test this, sometimes it takes too long to load.

[ ] - Create SQL script to apply EF Core in production
[ ] - Complete JS client
[ ] - Implement database seeder
[ ] - Test app when it is finished

I am leaving this PR as a WIP because I need a break. Though if you want to help, please feel free to collaborate and contribute your ideas or suggestions through my fork or anywhere😊

@Jklawreszuk
Could you send met a link / zip to the database dump you used and I can test it?

I am no longer working this PR, see https://github.com/stride3d/stride/pull/2261


Linker command failed error at XCode build time

Platform
Apple
Installed
CocoaPods
Version
0.9.0
Steps to Reproduce
Kotlin 2.0.20
project build.gradle.kts:
  kotlin("native.cocoapods") version "2.0.20" apply false
module build.gradle.kts:
 kotlin("native.cocoapods") version "2.0.20"
pod setup:
    cocoapods {
        version = "1.0"
        name = "SentryKMP"

        specRepos {
            url("https://github.com/Kotlin/kotlin-cocoapods-spec.git")
        }
        pod("Sentry") {
            version = "~> 8.25"
            linkOnly = true
            extraOpts += listOf("-compiler-option", "-fmodules")
        }
    }

gradle.properties:
kotlin.apple.deprecated.allowUsingEmbedAndSignWithCocoaPodsDependencies=true
Expected Result
Project builds successfully.
Actual Result
Getting this error when trying to build (only happens if I import the Sentry classes and use them in my code, but not otherwise):

Same problem, waiting for a solution

could you try cocoapods 8.36.0?
are you on xcode 16?

I am having the same issue with pod GoogleSignIn

kotlin:2.0.21, sentry-kotlin-multiplatform:0.10.0, same issue

podfile = project.file("../iosApp/Podfile")
add this file
# Uncomment the next line to define a global platform for your project
# platform :ios, '9.0'

target 'iosApp' do
  # Comment the next line if you don't want to use dynamic frameworks
  use_frameworks!
  platform :ios, '15.3' # The version should be the same as the one declared in the build.gradle.kts file
  pod 'composeApp', :path => '../composeApp'

  # Pods for iosApp
  # Others external dependencies used to import them in a Swift file
  # pod 'MQTTClient' # Not needed because I will use the lib in my Kotlin code and not in the Swift code

  # We need to have all pods at the SAME iOS version
  post_install do |installer|
    installer.pods_project.targets.each do |target|
      target.build_configurations.each do |config|
        config.build_settings["IPHONEOS_DEPLOYMENT_TARGET"] = "15.3"
      end
    end
  end
end

can run on iOS , but when I init sentry , get :
*** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[Sentry.SentryReplayOptions setMaskAllText:]: unrecognized selector sent to instance 0x3031609b0'
*** First throw call stack:
(0x18ed407cc 0x18c0132e4 0x18ee468c8 0x18ecdeb08 0x18ecde430 0x10b0bf024 0x10b0bf848 0x10b0bf934 0x1097b1f94 0x10b0a3268 0x10b0a3300 0x10b0a33ec 0x1097b1f94 0x10b0be6cc 0x10b0be758 0x10b0be844 0x1097b1f94 0x10b0a2780 0x104105348 0x10b0c0384 0x10b0a2620 0x10b0be5fc 0x10b0bfb64 0x10b0a31a8 0x10b0a3080 0x10b0b6c20 0x108e42158 0x1097b6134 0x10965bbc4 0x1097b6220 0x1098e74ac 0x10990cbb4 0x10985825c 0x1097b6220 0x10965bdfc 0x1097b6220 0x1098dff3c 0x109910308 0x10a4e4640 0x10a4e53cc 0x10a4e549c 0x1097b33b0 0x10a4e3950 0x10a4e37ec 0x10a513a50 0x10a4f9ee0 0x10a5e5444 0x10a56594c 0x10a55a88c 0x10a55d034 0x10a55d15c 0x1097b69c0 0x10a5b915c 0x10a5b91f4 0x10a5b931c 0x1097b69c0 0x10a5af29c 0x10a5ae0e0 0x10a5b8c9c 0x10a5b7e88 0x10a5ba4a0 0x1914b1688 0x1907e7c28 0x1907e77b4 0x19083e914 0x1907bd7c4 0x19164c6a0 0x18ed236e4 0x18ed11910 0x18ed10fd4 0x18ed10830 0x1dacf01c4 0x191876eb0 0x1919255b4 0x193479f98 0x19345a664 0x19345d490 0x108dd857c 0x108dd862c 0x1b46feec8)
libc++abi: terminating due to uncaught exception of type NSException

update Sentry to '8.39.0' can fix it
    cocoapods {
        version = "1.0"
        summary = "Some description for the Shared Module"
        homepage = "Link to the Shared Module homepage"
        // Make sure you use the proper version according to our Cocoa SDK Version Compatibility Table.
        ios.deploymentTarget = "15.3"
        podfile = project.file("../iosApp/Podfile")
        pod("Sentry") {
            version = "8.39.0"
            linkOnly = true
            extraOpts += listOf("-compiler-option", "-fmodules")
        }
        framework {
            baseName = "ComposeApp"
            isStatic = true
        }
    }

👍

can you provide a repro sample where it doesnt work?


Add a delete Thing button

Just click on the magical trash can and all your worries will go away! The function is located inside the main controller as "deleteThing(name)." Just give the name and it will be deleted.


Set chunk_size according to operator output row size

Enhancement
Current chunk_size is static, 16384 for pipeline engine, which has following limitations:

For plan that generate large row size, the chunk would become very large, and cause the memory allocation performance issues. Specifically, allocating a large chunk from tcmalloc is pretty slow, which must use the CentralCache.

In this solution, we would decide the chunk_size for different fragment.

For fragment which output small row size, use the default 16384 chunk_size
For fragment which output large row size, use a smaller chunk_size 4096.

Moreover, xxx

Since we have changed the chunk-size in pipeline from 16384 to 4096, current we don't need to dynamic decide the chunk size.