Hugging Face  'Noxilus/doctr-torch-parseq-german' model causes error

Bug description
When loading 'Noxilus/doctr-torch-parseq-german' model from hug i encounter error of size mismatch
Code snippet to reproduce the bug
    # Load a custom recognition model from huggingface hub
    reco_model = from_hub('Noxilus/doctr-torch-parseq-german')

    # You can easily plug in this models to the OCR predictor
    model = ocr_predictor(det_arch='db_resnet50', reco_arch=reco_model,pretrained=True)

Error traceback

RuntimeError                              Traceback (most recent call last)
 in <cell line: 5>()
11         # Load a custom recognition model from huggingface hub
12         #reco_model = from_hub('Noxilus/doctr-torch-parseq-german')
---> 13         reco_model = from_hub('Noxilus/doctr-torch-parseq-german')
14
15         # You can easily plug in this models to the OCR predictor
1 frames
/usr/local/lib/python3.10/dist-packages/doctr/models/factory/hub.py in from_hub(repo_id, **kwargs)
233     if is_torch_available():
234         state_dict = torch.load(hf_hub_download(repo_id, filename="pytorch_model.bin", **kwargs), map_location="cpu")
--> 235         model.load_state_dict(state_dict)
236     else:  # tf
237         repo_path = snapshot_download(repo_id, **kwargs)
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict, assign)
2151
2152         if len(error_msgs) > 0:
-> 2153             raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
2154                                self.class.name, "\n\t".join(error_msgs)))
2155         return _IncompatibleKeys(missing_keys, unexpected_keys)
RuntimeError: Error(s) in loading state_dict for PARSeq:
size mismatch for head.weight: copying a param with shape torch.Size([196, 384]) from checkpoint, the shape in current model is torch.Size([84, 384]).
size mismatch for head.bias: copying a param with shape torch.Size([196]) from checkpoint, the shape in current model is torch.Size([84]).
size mismatch for embed.embedding.weight: copying a param with shape torch.Size([198, 384]) from checkpoint, the shape in current model is torch.Size([86, 384]).
Environment
Collecting environment information...
DocTR version: v0.8.1
TensorFlow version: 2.15.0
PyTorch version: 2.2.1+cu121 (torchvision 0.17.1+cu121)
OpenCV version: 4.8.0
OS: Ubuntu 22.04.3 LTS
Python version: 3.10.12
Is CUDA available (TensorFlow): No
Is CUDA available (PyTorch): No
CUDA runtime version: 12.2.140
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.6
Deep Learning backend
is_tf_available: False
is_torch_available: True

Hi @yumikim381 :wave:,
I checked this and it looks like that the user made an mistake while uploading the model (the vocab in the config file seems to differ to the one which was used for training)
Unfortunately we can't do anything about this the owner of this shared model needs to fix it.
You can use:
tilman-rassy/doctr-crnn-vgg16-bn-fascan-v1 (trained on german + french vocab)
or
Felix92/doctr-torch-parseq-multilingual-v1 (was fine tuned on multilingual vocab https://mindee.github.io/doctr/modules/datasets.html#supported-vocabs)
instead.
Or train your own model you can generate some synth data with: https://github.com/felixdittrich92/synthtiger/tree/doctr-modified  (It's not really clean yet but it works :sweat_smile: I have planned to clean and simplify this in the upcoming weeks)

Thanks a lot!!

So i think we can close this :)


TypeScript version not shown

VSCode Version: Code - Insiders 1.12.0-insider (0bec115779b11a9d78bdb24ad7918c22524ea2ac, 2017-04-28T18:57:52.518Z)
OS Version: Windows_NT ia32 10.0.15063

Steps to Reproduce:

Open JavaScript file
TypeScript version is not shown (Codelenses neither)

Yes, we only show the TypeScript version in that status bar in TS files. The reasoning is that for the majority for JS files, the typescript version does not matter. You can still view the active version by running the Select TypeScript Version command
As for the code lenses, these are off by default. You must set "typescript.referencesCodeLens.enabled": true to enable them

I think this should be re-open : as TypeScript 2.3 introduced checkJs option, I have a project in full JS (no TS) but using TypeScript checking. And with the current behavior, there is no simple way to change the TypeScript version used by Code (for example to use a typescript@next release). I need to create a fake .ts file to be able to change the TS version...

@cyrilletuzi You can also run TypeScript: Select TypeScript Version in the command palette. This command is effectively the same as clicking on the version in the status bar


bugfix: TextMatchFilterOptimizer grouping for inner compound query

I missed this edge case when adding TextMatchFilterOptimizer.
For a query such as text_match(col1, 'aaa') AND text_match(col1, 'bbb OR ccc'), this would have previously been optimized to text_match(col1, '(foo AND bar OR baz)', which would return unexpected results.
Applying grouping for each filter (instead of at the merged filter level) produces an faithful translation to text_match(col1, '(foo) AND (bar OR baz)'.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 0.00%. Comparing base (59551e4) to head (4292534).
Report is 1237 commits behind head on master.

:exclamation:  There is a different number of reports uploaded between BASE (59551e4) and HEAD (4292534). Click for more details.
HEAD has 48 uploads less than BASE

Flag
BASE (59551e4)
HEAD (4292534)

integration
7
2

integration2
3
2

temurin
12
2

java-21
7
2

skip-bytebuffers-true
3
1

skip-bytebuffers-false
7
1

unittests
5
0

unittests1
2
0

java-11
5
0

unittests2
3
0

integration1
2
0

custom-integration1
2
0

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #14299       +/-   ##
=============================================
- Coverage     61.75%    0.00%   -61.76%
=============================================
  Files          2436        3     -2433
  Lines        133233        6   -133227
  Branches      20636        0    -20636
=============================================
- Hits          82274        0    -82274
+ Misses        44911        6    -44905
+ Partials       6048        0     -6048

Flag
Coverage Δ

custom-integration1
?

integration
0.00% <ø> (-0.01%)
:arrow_down:

integration1
?

integration2
0.00% <ø> (ø)

java-11
?

java-21
0.00% <ø> (-61.63%)
:arrow_down:

skip-bytebuffers-false
0.00% <ø> (-61.75%)
:arrow_down:

skip-bytebuffers-true
0.00% <ø> (-27.73%)
:arrow_down:

temurin
0.00% <ø> (-61.76%)
:arrow_down:

unittests
?

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


fix: correctly set last resub time

The avlSubscriptionSchema should include the new lastResubscriptionTime property as z.string().nullish()

It should already be there from a previous branch


Setup not working on Neovim when installed with MacOS Homebrew

When installing Nvim via MacOS Homebrew package manager it seems like the setup file will put the config files in the incorrect location.
Anyway this could be fixed?

This is actually my bad. The nvim folder was not created inside the .config folder. It would be great if the script could create the folder if it doesn't already exist!

Good idea! I'll add that in the next update.


Inquiry about the default config of PTv3

Hi Xiaoyang. I find that the default config here of PTv3 in this repo is a little bit different from the original paper (stages and dec depth).

Does the current config get better performance than the paper? Thx.

Oh, yes. But trust me, the parameter doesn't matter so much. The scale of a model should go with the scale of data.

Got that. Agree with you. I find that the result on ScanNet v2 Val set is just 0.1% larger than the paper. lol


docs: update readme following recent project changes

This PR makes a few updates to the QL language docs readme.rst file following recent changes to the organisation of the sphinx projects:

ql-training-rst renamed to ql-training
training project incorporated into learn-ql
template.rst added
brief mention of sphinx link checker

These changes all LGTM. Thanks for the update


macOS: Cannot save changes to "Client settings"

macOS: 12.7.5 Intel
MusicAssistant companion app: 0.0.45
MusicAssistant: 2.0.4
in the macOS companion app there is no way to save/apply changes to the "Client settings".
I was trying to change the "Audio output device" but there isn't any save button.
When I change tab and then come back to the Client settings any changes I made revert to the default settings.
Thanks

Ok thanks for pointing it out! I can maybe take a look later toady

thanks for the answer.
I was trying the new version 0.0.53, it seems the bug is still present.

Yeah sorry, I haven't had time to fix it yet sorry.

don't worry, thanks anyway for all your time and work in this project.


refactor IIIFImage logic

also: removing unnecessary height value

I think height might have been necessary as now the background color placeholder doesn't have the expected image height? (Or it's something else!)

And I'm seeing a few pixels of that background color at the bottom of all the images


Partial fix for #1682

Mostly test cases that fails in the current 4.4.x branch, and a quick fix to the problem in #1682.
testAggregateEventsAreReadFromFirstSequenceNumberHistoricOnly is for completeness, it also passes with the current implementation.
The fix feels a bit of a hack, but I don't know the code well enough at this point. It could be that the ConcatenatingDomainEventStream should contain the first sequence number we tried to fetch as a member, and instead apply the fix in the private nextSequenceNumber()  method.
Something like:
        private final long firstSequenceNumber;

         // ...

        private long nextSequenceNumber() {
            Long lastSequenceNumber = historic.getLastSequenceNumber();
            return lastSequenceNumber == null ? firstSequenceNumber : lastSequenceNumber + 1;
        }

I can change the PR to include the above instead, just le me know.
--
Harald K

I just updated the PR as suggested, as it feels a lot cleaner.
--
Harald K

I just updated the PR as suggested, as it feels a lot cleaner.
--
Harald K

I agree your current solution is cleaner than the previous one @haraldk.
There is one thing which still bothers me though.
What I see happening with this fix, is assuring that the active event stream starts at the given firstSequenceNumber, if the historic stream didn't contain anything.
This means that currently if you would get a snapshot and then move to your active stream, that right now the active stream would start at position zero.
Thus, you would not only be handling the snapshot, but also all the events.
However, your original issue states the following:

Debugging shows that for some aggregates, we no longer receive the events after the snapshot event from AbstractEventStore.readEvents, we get a stream containing only the single snapshot event (which again causes massive problems later on, when trying to add new events).

As well as stating what I would assume as well:

I would expect the incorrectly low number to return more or even duplicate events, not zero events.

I was wondering whether you have tested out this fix against your environment too.
I see this solves a problem, but I would want to make sure it solves the situation you are describing too.

I agree your current solution is cleaner than the previous one @haraldk.
There is one thing which still bothers me though.
What I see happening with this fix, is assuring that the active event stream starts at the given firstSequenceNumber, if the historic stream didn't contain anything.
This means that currently if you would get a snapshot and then move to your active stream, that right now the active stream would start at position zero.
Thus, you would not only be handling the snapshot, but also all the events.
However, your original issue states the following:

Debugging shows that for some aggregates, we no longer receive the events after the snapshot event from AbstractEventStore.readEvents, we get a stream containing only the single snapshot event (which again causes massive problems later on, when trying to add new events).

As well as stating what I would assume as well:

I would expect the incorrectly low number to return more or even duplicate events, not zero events.

I was wondering whether you have tested out this fix against your environment too.
I see this solves a problem, but I would want to make sure it solves the situation you are describing too.

Yes,
We have tested that the fix does indeed solve our problem, as fas as we can see. However, I still think it is strange that it does (it's more of an optimization than a fix in this case)...
My current theory is that the case when firstSequenceNumber is 0 (or just less than the lowest sequence number for that aggregate in the active storage?) it triggers another problem in JdbcEventStorageEngine (or Batching...) that makes it return no events at all... But I have yet to prove this.
--
Harald K

Yes,
We have tested that the fix does indeed solve our problem, as fas as we can see. However, I still think it is strange that it does (it's more of an optimization than a fix in this case)...
My current theory is that the case when firstSequenceNumber is 0 (or just less than the lowest sequence number for that aggregate in the active storage?) it triggers another problem in JdbcEventStorageEngine (or Batching...) that makes it return no events at all... But I have yet to prove this.
--
Harald K

Could it be this:
https://github.com/AxonFramework/AxonFramework/blob/af170661f33f7973e7d3b99cb16807d53fc33f6b/eventsourcing/src/main/java/org/axonframework/eventsourcing/eventstore/jdbc/statements/JdbcEventStorageEngineStatements.java#L330
    public static PreparedStatement readEventDataForAggregate(Connection connection, EventSchema schema,
                                                              String identifier, long firstSequenceNumber,
                                                              int batchSize) throws SQLException {
        final String sql =
                "SELECT " + schema.trackedEventFields() + " FROM " + schema.domainEventTable() + " WHERE " + schema
                        .aggregateIdentifierColumn() + " = ? AND " + schema.sequenceNumberColumn() + " >= ? AND "
                        + schema.sequenceNumberColumn() + " < ? ORDER BY "
                        + schema.sequenceNumberColumn() + " ASC";
        PreparedStatement statement = connection.prepareStatement(sql);
        statement.setString(1, identifier);
        statement.setLong(2, firstSequenceNumber);
        statement.setLong(3, firstSequenceNumber + batchSize);
        return statement;
    }

If firstSequenceNumber here incorrectly is 0, we will only fetch sequence numbers between 0 and batch size... And as we already have more than batch size in historic, active will have no matches here...
--
Harald K

Could it be this:
https://github.com/AxonFramework/AxonFramework/blob/af170661f33f7973e7d3b99cb16807d53fc33f6b/eventsourcing/src/main/java/org/axonframework/eventsourcing/eventstore/jdbc/statements/JdbcEventStorageEngineStatements.java#L330
    public static PreparedStatement readEventDataForAggregate(Connection connection, EventSchema schema,
                                                              String identifier, long firstSequenceNumber,
                                                              int batchSize) throws SQLException {
        final String sql =
                "SELECT " + schema.trackedEventFields() + " FROM " + schema.domainEventTable() + " WHERE " + schema
                        .aggregateIdentifierColumn() + " = ? AND " + schema.sequenceNumberColumn() + " >= ? AND "
                        + schema.sequenceNumberColumn() + " < ? ORDER BY "
                        + schema.sequenceNumberColumn() + " ASC";
        PreparedStatement statement = connection.prepareStatement(sql);
        statement.setString(1, identifier);
        statement.setLong(2, firstSequenceNumber);
        statement.setLong(3, firstSequenceNumber + batchSize);
        return statement;
    }

If firstSequenceNumber here incorrectly is 0, we will only fetch sequence numbers between 0 and batch size... And as we already have more than batch size in historic, active will have no matches here...
--
Harald K

Yep, I think you got it there @haraldk. Key was you sharing the storage engine being used really.
I was delving into the AbstractEventStore end, for which I just created a couple of tests to validate the ConcatenatingDomainEventStream (not the private class in the SequenceEventStorageEngine, but the one used in the AbstractEventStore).
This landed me up with nothing, but in the mean time, you provided this thought.
The culprit this time around is the BatchingEventStorageEngine.EventStreamSpliterator.
The tryAdvance method will be in charge of invoking your storage solution batch wise.
If the result returned an empty set of 0, tryAdvance will return false.
This means that the EventStorageEngine implementation (this wouldn't be different between JPA, JDBC (as in your case) or Mongo afaik) would no be invoked further, as there were not events present.
This also answers why increasing the batch-size resolved your problem.
Great, thanks for all the info here @haraldk!

Yep, I think you got it there @haraldk. Key was you sharing the storage engine being used really.
I was delving into the AbstractEventStore end, for which I just created a couple of tests to validate the ConcatenatingDomainEventStream (not the private class in the SequenceEventStorageEngine, but the one used in the AbstractEventStore).
This landed me up with nothing, but in the mean time, you provided this thought.
The culprit this time around is the BatchingEventStorageEngine.EventStreamSpliterator.
The tryAdvance method will be in charge of invoking your storage solution batch wise.
If the result returned an empty set of 0, tryAdvance will return false.
This means that the EventStorageEngine implementation (this wouldn't be different between JPA, JDBC (as in your case) or Mongo afaik) would no be invoked further, as there were not events present.
This also answers why increasing the batch-size resolved your problem.
Great, thanks for all the info here @haraldk!


Unittest für die getUserCoins backend route erstellt

add getUserCoins unit test
fixes #183
:ballot_box_with_check: Definition of Done checklist

[x] Frontend

[x] Are all functions implemented?
[x] Are there front-end tests that test the appearance of the mockup?
[x] Do the associated components look like the mockup?

[x] Backend

[x] Are all necessary routes documented via swagger?
[x] Are all necessary methods tested via Jest (positive and negative tests)?

[x] Is the user story finished to such an extent that no further work is required after the pull request (excluding hotfixes and adjustments)?

guter Hinweis, das Coins return habe ich angepasst
Das hoch und heruntersetzen würde ich dann mit dem Issue #190 betrachten


[FEAT] Redis Stack Data Provider

I'd like to try to connect Redis Stack to Refine. Redis Stack has capabilities of document store and search engine in one database.
The manual says:

A data provider makes HTTP requests and returns response data back using predefined methods.

I'd like to skip the HTTP layer for now. The content is all created and consumed through the Refine app and an API is not needed at the moment.
Any feedback?

Hey @amiorin ,
In this case, I think you can use Next.js. Is there any support we can provide?


modules/components in aloxide

list out modules/components in aloxide

This is an ongoing list. We can update our document base on this list.
/cc @quocle108 @flowersinthesand

moduels
description

@aloxide/abstraction
provide tool for generating smart-contract

@aloxide/bridge
provide adapter for integrating with blockchain

@aloxide/api-gateway
provide an API template for the back-end side

@aloxide/sdk
TBD

@aloxide/demux
TBD

What's the role of @aloxide/demux?

What's the role of @aloxide/demux?

It is a package helping api-gateway up to date with blockchain. I am not sure yet :).

I will close this issue, and refer to it later when we need to update some wikis.


Fix bug on fixed_last_octet resources command

@oNaiPs seems now we're using AssignFromString

@pablosole fixed and added another fix on a new commit


Adding explicit CPU conversion in order to work with torch 1.0.0 and GPU

added explicitly the conversion to CPU necessary when working on GPU before converting a torch tensor to a numpy array.   Use Tensor.cpu() to copy the tensor to host memory first.

.eggs/torch-1.0.0-py3.7-linux-x86_64.egg/torch/tensor.py:450: TypeError

Sorry about that, I had access issues that prevented me from committing directly from my AWS instance where I ran the tests on GPU so I forgot a line when I applied the changes on my local machine but this time I directly downloaded the modified files from my AWS instance on which tests on GPU with torch==1.0.0 passed

Codecov Report

Merging #246 into master will not change coverage.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master     #246   +/-   ##
=======================================
  Coverage   91.26%   91.26%
=======================================
  Files          30       30
  Lines        2302     2302
=======================================
  Hits         2101     2101
  Misses        201      201

Impacted Files
Coverage Δ

scvi/inference/posterior.py
95.43% <100%> (ø)
:arrow_up:

scvi/inference/annotation.py
85.92% <100%> (ø)
:arrow_up:

scvi/inference/trainer.py
87.03% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f3eef4f...d5b6682. Read the comment docs.

Thanks Jules!


Apply for adding MegEngine to list of projects using the theme

Congrats~pydata-sphinx-theme is becoming better and better.
Our Project MegEngine (A fast, scalable, and easy-to-use deep learning framework, with auto-differentiation) is using this theme now. [The Doc Site] Hope to be added to your user list.
As you can see, we add a simple version/language switcher. (https://github.com/MegEngine/pydata-sphinx-theme/pull/1) Are you interested in this feature?

Very cool! Yep feel free to create a PR so we can add it to the list ✨
For the version switcher, that looks nice, thanks very much! There have been a few other discussions around the version switcher, want to discuss there? https://github.com/pydata/pydata-sphinx-theme/issues/23 is the main discussion, and there's one proposed implementation here #276


[BUG] OpenSearchISMPolicy doesn't work

I created a OpenSearchISMPolicy according to the current spec, but it isn't created in opensearch (error in logs is the object has been modified; please apply your changes to the latest version and try again).
Looks like this kind doesn't work at all (check also https://github.com/opensearch-project/opensearch-k8s-operator/discussions/696 this)
apiVersion: opensearch.opster.io/v1
kind: OpenSearchISMPolicy
metadata:
  name: rollover
  namespace: infra
spec:
  opensearchCluster:
    name: opensearch
  policyId: default
  description: Rollover ISM policy
  ismTemplate:
    priority: 100
    indexPatterns:
      - "apps*"
      - "kube*"
      - "node*"
      - "ingress*"
  errorNotification:
    channel: slack
    destination:
      slack:
        url: "${SLACK_ALERTS_WEBHOOK_URL}"
    messageTemplate:
      source: "The index {{ctx.index}} failed during ISM policy execution"
  defaultState: hot
  states:
    - name: hot
      actions:
        - rollover:
            minIndexAge: 10d
            minPrimaryShardSize: 10gb
        - indexPriority:
            priority: 100
      transitions:
        - stateName: warm
          conditions:
            minIndexAge: 7d
            minRolloverAge: 7d
    - name: warm
      actions:
        - indexPriority:
            priority: 50
      transitions:
        - stateName: delete
          conditions:
            minIndexAge: 20d
    - name: delete
      transitions: []
      actions:
        - delete: {}

logs:
{"level":"info","ts":"2024-02-20T13:18:33.780Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"infra"},"namespace":"infra","name":"opensearch","reconcileID":"7e721d68-7a74-4aa9-ae84-add870efee35","cluster":{"name":"opensearch","namespace":"infra"}}
{"level":"info","ts":"2024-02-20T13:18:33.799Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"infra"},"namespace":"infra","name":"opensearch","reconcileID":"7e721d68-7a74-4aa9-ae84-add870efee35","interface":"transport"}
{"level":"info","ts":"2024-02-20T13:18:33.799Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"infra"},"namespace":"infra","name":"opensearch","reconcileID":"7e721d68-7a74-4aa9-ae84-add870efee35","interface":"http"}
{"level":"info","ts":"2024-02-20T13:18:57.218Z","msg":"Reconciling OpensearchISMPolicy","controller":"opensearchismpolicy","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchISMPolicy","OpenSearchISMPolicy":{"name":"rollover","namespace":"infra"},"namespace":"infra","name":"rollover","reconcileID":"cd48d62d-3287-4063-811f-54a910d41b1f","tenant":{"name":"rollover","namespace":"infra"}}
{"level":"info","ts":"2024-02-20T13:18:57.227Z","msg":"Reconciling OpensearchIndexTemplate","controller":"opensearchindextemplate","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchIndexTemplate","OpensearchIndexTemplate":{"name":"all","namespace":"infra"},"namespace":"infra","name":"all","reconcileID":"67058e28-82d7-4951-ac6f-0c8235a3d7b9","indextemplate":{"name":"all","namespace":"infra"}}
{"level":"info","ts":"2024-02-20T13:18:57.350Z","msg":"Reconciling OpensearchISMPolicy","controller":"opensearchismpolicy","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchISMPolicy","OpenSearchISMPolicy":{"name":"rollover","namespace":"infra"},"namespace":"infra","name":"rollover","reconcileID":"a627ee69-6d8e-4aec-b9e0-53e4cc0a09b7","tenant":{"name":"rollover","namespace":"infra"}}
{"level":"error","ts":"2024-02-20T13:18:57.363Z","msg":"Reconciler error","controller":"opensearchismpolicy","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchISMPolicy","OpenSearchISMPolicy":{"name":"rollover","namespace":"infra"},"namespace":"infra","name":"rollover","reconcileID":"a627ee69-6d8e-4aec-b9e0-53e4cc0a09b7","error":"Operation cannot be fulfilled on opensearchismpolicies.opensearch.opster.io \"rollover\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"info","ts":"2024-02-20T13:18:57.363Z","msg":"Reconciling OpensearchISMPolicy","controller":"opensearchismpolicy","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchISMPolicy","OpenSearchISMPolicy":{"name":"rollover","namespace":"infra"},"namespace":"infra","name":"rollover","reconcileID":"2deb7560-cf4d-4e6c-a7fa-76143ec10945","tenant":{"name":"rollover","namespace":"infra"}}
{"level":"info","ts":"2024-02-20T13:18:57.422Z","msg":"Reconciling OpensearchIndexTemplate","controller":"opensearchindextemplate","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchIndexTemplate","OpensearchIndexTemplate":{"name":"all","namespace":"infra"},"namespace":"infra","name":"all","reconcileID":"3b6836b9-10f8-41a9-9ec0-63f0b41450bb","indextemplate":{"name":"all","namespace":"infra"}}
{"level":"info","ts":"2024-02-20T13:18:57.440Z","msg":"Reconciling OpensearchISMPolicy","controller":"opensearchismpolicy","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchISMPolicy","OpenSearchISMPolicy":{"name":"rollover","namespace":"infra"},"namespace":"infra","name":"rollover","reconcileID":"cdf2e629-0b1a-4ff8-a8df-fc4613491081","tenant":{"name":"rollover","namespace":"infra"}}

docker image: opensearchproject/opensearch-operator:2.5.1

Hi, I'm trying to do the same, but I have some doubts.
After create the OpenSearchISMPolicy resource I need to reference this in cluster configurations?
How this is applying in the cluster?

ISMPolicy references cluster via
spec:
  opensearchCluster:
    name: opensearch

OK, I am thar reference but I can't see my policy in dashboard

[Triage]
Looks like this works for @cthtrifork based on the PR that was recently merged https://github.com/opensearch-project/opensearch-k8s-operator/pull/750, @cthtrifork can you please help here.
Thanks

Hmm I am unsure if my fix is related and it sounds like a kubernetes reconcile error. Can we create a beta release of the reconciler and I can help with testing and investigation.
If a official beta release is not possible, then a guide for building and creating the docker image which you could push to a private registry.


Refactored routing for front-end, added to docs.

Summary
Refactored the routing to allow for deep-linking.  This will be a breaking change between v1 and v2, as we will be supporting more identity providers.
Description for the changelog
Enabled deep-linking for threat model view/edit operations
Other info
Documentation will be available at https://www.threatdragon.com/docs/usage/collaboration/linking.html after the merge/automated deploy is complete.

Codecov Report

Merging #185 (b2d32a7) into v2-development (d09e256) will increase coverage by 0.07%.
The diff coverage is n/a.

@@                Coverage Diff                 @@
##           v2-development     #185      +/-   ##
==================================================
+ Coverage           97.75%   97.83%   +0.07%
==================================================
  Files                  60       63       +3
  Lines                 847      876      +29
  Branches               14       20       +6
==================================================
+ Hits                  828      857      +29
  Misses                 19       19

Impacted Files
Coverage Δ

src/router/index.js
100.00% <0.00%> (ø)

src/components/Navbar.vue
100.00% <0.00%> (ø)

src/store/modules/threatmodel.js
100.00% <0.00%> (ø)

src/views/Branch.vue

src/views/Threatmodel.vue

src/service/github.provider.js

src/views/ThreatmodelEdit.vue

src/service/session.provider.js

src/views/ThreatmodelSelect.vue

src/service/providers.js

... and 12 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d09e256...b2d32a7. Read the comment docs.

Great to see this all happening @lreading


Fix ipfamily classification

The net.To4 could convert an IPv6 address to an IPv4 address.
Using To4 could cause IPv6 addresses wrongly classified as IPv4 ones.
More details in https://github.com/golang/go/blob/189c6946f598dc668946499e4179775c06295f9d/src/net/ip.go#L209-L222
Signed-off-by: Hanlin Shi<EMAIL_ADDRESS>

cool, thx for the info and fix :)
/lgtm
/approve

@hanlins I'm reverting as it broke IPSet:
--- FAIL: ExampleIPSetAdd (0.00s)
got:
[]
[<IP_ADDRESS> <IP_ADDRESS> <IP_ADDRESS> <IP_ADDRESS> ::1 ::2 ::3 ::4]
want:
[<IP_ADDRESS> <IP_ADDRESS> <IP_ADDRESS> <IP_ADDRESS>]
[::1 ::2 ::3 ::4]
FAIL
FAIL	sigs.k8s.io/kpng/pkg/api/localnetv1	0.003s


default to weights_only=True for safer loading

Given that people may be sharing around models, weights_only=True may be a safer way to ensure only weights are loaded as it is not a default torch option yet.

@JarodMica Thanks, not sure if something like 'step' in the dict will be dropped or not.
may you could help us check, if is fine with normal training process, will just merge~

@JarodMica Thanks, not sure if something like 'step' in the dict will be dropped or not. may you could help us check, if is fine with normal training process, will just merge~

Sure, I'll check it out and report back after the epoch on my latest training run finishes through

Alright, so training is running with weight_only=True.  It seems to be chugging along as normal.
However, two things to note: I have not tried to inference with a model out of this training yet and I am running this on an older version of the main branch, though, I don't think any of the training functions have changed.


Browser back button not working in Firefox

Hi,
I'm currently looking into an issue where the browser's back button does not work as expected when accessing Lona views using the Firefox browser. It works as expected when using Chrome/-ium but only seems to create a reload in Firefox.
Has anyone observed the same problem and/or found a way to solve it? I'm mostly puzzled by this only occurring in one browser.

Hi @m-reichle!
Interesting! Do I have to do something specific to trigger your problem? Do you have a reproducer for me?

Hi @m-reichle. I can reproduce the problem and it is a bug for sure. The history behavior is wrong, but both browsers behave the same for me. I debugged it already, and I think the problem is that I invoke pushState too often. So the history is actually "correct", but I add too many entries to it. I hope to have a solution ready this week.
Thanks for reporting! :)

@m-reichle: can you check if #552 fixes your issue?

Hi,
I've tested it both with my test case and our in-house application and it seems to have fixed the issue.
Thank you for taking care of this!

@m-reichle: Great! Thanks for testing :)
No problem.
I will cleanup the PR a bit and merge it afterwards. I assume you need a release right?

@m-reichle: The fix is available in 1.16.2. Thanks for contributing :)

Awesome! Thanks a lot!!


start vllm.entrypoints.api_server model vicuna-13b-v1.3  error: Fatal Python error: Bus error

err detail
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at https://github.com/huggingface/transformers/pull/24565 (RayWorker pid=3554640) *** SIGBUS received at time=1692931519 on cpu 133 *** (RayWorker pid=3554640) PC: @     0x7f60c5616291  (unknown)  __memset_avx2_erms (RayWorker pid=3554640)     @     0x7f60c60a8cf0  (unknown)  (unknown) (RayWorker pid=3554640) [2023-08-25 10:45:19,168 E 3554640 3555230] logging.cc:361: *** SIGBUS received at time=1692931519 on cpu 133 *** (RayWorker pid=3554640) [2023-08-25 10:45:19,168 E 3554640 3555230] logging.cc:361: PC: @     0x7f60c5616291  (unknown)  __memset_avx2_erms (RayWorker pid=3554640) [2023-08-25 10:45:19,168 E 3554640 3555230] logging.cc:361:     @     0x7f60c60a8cf0  (unknown)  (unknown) (RayWorker pid=3554640) Fatal Python error: Bus error (RayWorker pid=3554640)  2023-08-25 10:45:21,197 WARNING worker.py:2037 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffff781cee214007f7ef2aa9c68501000000 Worker ID: 0b9f32035f3bf5887270902660bc1c930f4fde24efc863eb33869403 Node ID: 60ec9d72124b5f6b8778a26b6fc79798f3791e315e57cac00bddb1ec Worker IP address: <IP_ADDRESS> Worker port: 37577 Worker PID: 3554643 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors. Traceback (most recent call last):   File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main     return _run_code(code, main_globals, None,   File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code     exec(code, run_globals)   File "/usr/local/lib64/python3.9/site-packages/vllm/entrypoints/api_server.py", line 78, in <module>     engine = AsyncLLMEngine.from_engine_args(engine_args)   File "/usr/local/lib64/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 232, in from_engine_args     engine = cls(engine_args.worker_use_ray,   File "/usr/local/lib64/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 55, in __init__     self.engine = engine_class(*args, **kwargs)   File "/usr/local/lib64/python3.9/site-packages/vllm/engine/llm_engine.py", line 99, in __init__     self._init_workers_ray(placement_group)   File "/usr/local/lib64/python3.9/site-packages/vllm/engine/llm_engine.py", line 170, in _init_workers_ray     self._run_workers(   File "/usr/local/lib64/python3.9/site-packages/vllm/engine/llm_engine.py", line 474, in _run_workers     all_outputs = ray.get(all_outputs)   File "/usr/local/lib64/python3.9/site-packages/ray/_private/auto_init_hook.py", line 24, in auto_init_wrapper     return fn(*args, **kwargs)   File "/usr/local/lib64/python3.9/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper     return func(*args, **kwargs)   File "/usr/local/lib64/python3.9/site-packages/ray/_private/worker.py", line 2526, in get     raise value ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.         class_name: RayWorker         actor_id: 781cee214007f7ef2aa9c68501000000         pid: 3554643         namespace: 78788d7f-d9c6-4ea7-8992-0debc2720932         ip: <IP_ADDRESS> The actor is dead because its worker process has died. Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.

Hi @luefei, could you share your environment (e.g., OS, CPU, CPU RAM, GPU)?

See here: https://github.com/NVIDIA/nccl-tests/issues/143
If you're running this in a container, it could potentially be due to /dev/shm being too small. In that case, it could be rectified by adding --shm-size 10.24g when you run the container.

I also have the same problem. The inference works on 4 gpus, however when I use 8 gpus, I got the identical error

Using --ipc=host fixed the issue for me. See vLLM docs here: https://docs.vllm.ai/en/latest/serving/deploying_with_docker.html

I also have the same problem. The inference works on 4 gpus, however when I use 8 gpus, I got the identical error

I also have the same problem. The inference works on 2 gpus however when I use 8 gpus in container


Question: Usage without autoPageviews: true

Hi there! I wanted to use Plausible in a Nuxt project and exclude some of my routes from being automatically tracked as pageview.
Since this is currently not possible with the plausible-tracker, I thought of setting autoPageviews: false and manually tracking them, excluding the paths I don't want to manually.
But where would be the best option to do that? In a Nuxt client middleware? Or in a route watcher in app.vue? Or maybe the page:finish hook? From the implementation of this nuxt module, it wasn't super clear for me on how to do it.
Appreciate your input!

Hey @madebyfabian, I think the easier way is to add manually the tracker in each page you want to track using https://github.com/nuxt-modules/plausible#usetrackpageview
<script setup>
useTrackPageview()
</script>

See usage in the UnJS website: https://github.com/unjs/website/blob/main/pages/index.vue#L21
Do not hesite if you have more question!

Thanks for your answer! I think there is now a native way to do that in the Plausible settings directly: https://plausible.io/docs/top-pages#block-traffic-from-specific-pages-or-sections

Closing this, as the issue seems to be resolved. 🏁


TwitterStatusMessage.Text not set in Extended mode

When using TwitterTweetMode.Extended, full_text in the JSON response is not mapped to TwitterStatusMessage.Text. (Or is this mode not fully supported yet?)
https://github.com/abjerner/Skybrud.Social.Twitter/blob/97a2967f0ef44f840f98d831a808f848b5e4964c/src/Skybrud.Social.Twitter/Models/Statuses/TwitterStatusMessage.cs#L98

@EvanMulawski Thanks a lot for reporting 👍
That is an oversight. I will look into pushing a fix ;)
Meanwhile, you can access the full text through tweet.JObject.GetString("full_text").

Thanks! This is my current workaround:
foreach (var statusMessage in statuses.Body)
{
    statusMessage.GetType().GetProperty(nameof(statusMessage.Text)).SetValue(statusMessage, statusMessage.JObject.Value<string>("full_text"), null);
}

@EvanMulawski I pushed a new release yesterday, which adds the FullText property.
Twitter's feature with extended tweets is implemented a bit weird in the way that text is omitted in the JSON for extended tweets, similar to how the full_text property is omitted when not using extended mode.
For the models in Skybrud.Social, I'm trying to follow the JSON structure, so similarly the Text property will be null in Skybrud.Social when using extended mode.
Usage would therefore be something like:
if (tweet.HasFullText)
{
<EMAIL_ADDRESS>}
else
{
<EMAIL_ADDRESS>}

or something like:
<pre>@(tweet.FullText ?? tweet.Text)</pre>


Error from auto-detecting Mr. Steam steamlinx gecko unit.

Version of the custom_component
Version 0.0.5 - from manifest.json
Version 0.0.1 - from const.py
pulled from master (default) branch at github.
Configuration
Integration added through UI, no other configuration done.  All options still enabled (binary sensor, climate, light, sensor, switch)

Describe the bug
Added the gecko-home-assistant custom component.  Enabled the integration. It auto detected the Mr. Steam Steamlinux Gecko unit, which isn't currently handled by Geckolib.  See Geckolib issue: https://github.com/gazoodle/geckolib/issues/6
Note: when I added the integration to Home Assistant the Gecko unit for the hot tub wasn't connected yet.
I'm not expecting Home Assistant support for the Mr. Steam yet, just looking to avoid the error from the auto-detection of the Mr. Steam unit.
Will report back what happens when integration/hass restarted with in.touch2 (hot tub) Gecko connected.
Debug log

Logger: custom_components.gecko
Source: util/async_.py:140
Integration: Gecko (documentation, issues)
First occurred: 1:29:13 PM (1 occurrences)
Last logged: 1:29:13 PM
Exception during entry setup

Traceback (most recent call last):
  File "/config/custom_components/gecko/__init__.py", line 45, in async_setup_entry
    facade = locator.get_spa_from_identifier(spa_identifier).get_facade(False)
  File "/usr/local/lib/python3.8/site-packages/geckolib/spa.py", line 46, in get_facade
    facade = GeckoFacade(GeckoSpa(self).start_connect())
  File "/usr/local/lib/python3.8/site-packages/geckolib/spa.py", line 73, in __init__
    GeckoSpaPack.__init__(self)
  File "/usr/local/lib/python3.8/site-packages/geckolib/driver/spapack.py", line 15, in __init__
    GeckoSpaPack.download_if_needed()
  File "/usr/local/lib/python3.8/site-packages/geckolib/driver/spapack.py", line 32, in download_if_needed
    GeckoSpaPack.download()
  File "/usr/local/lib/python3.8/site-packages/geckolib/driver/spapack.py", line 39, in download
    response = http.request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/request.py", line 74, in request
    return self.request_encode_url(
  File "/usr/local/lib/python3.8/site-packages/urllib3/request.py", line 96, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/usr/local/lib/python3.8/site-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.8/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1263, in _send_request
    self.putrequest(method, url, **skips)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 214, in putrequest
    return _HTTPConnection.putrequest(self, method, url, *args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/util/async_.py", line 151, in protected_loop_func
    check_loop()
  File "/usr/src/homeassistant/homeassistant/util/async_.py", line 140, in check_loop
    raise RuntimeError(
RuntimeError: I/O must be done in the executor; Use `await hass.async_add_executor_job()` at custom_components/gecko/__init__.py, line 45: facade = locator.get_spa_from_identifier(spa_identifier).get_facade(False)

This is the Home Assistant logger output.  If you need me to turn up debug logging, or give logging from the geckolib shell I can do that.

Have you tried to manually download and install the SpaPackStrict.xml file and put it in the config directory?
Your error looks to be related to the async download issue.

Thats fine.  Perfectly understand.   I've already fixed several.
Just giving back.
Krs
Mark

An update, after disconnecting the Steamlinx Gecko unit, deleting and re-adding the Gecko integration, and restarting Home Assistant I'm still seeking the same "Failed to set up" error.   Judging by the other pen issues with the same exception being thrown RuntimeError: I/O must be done in the executor;  I now need to look at the other fixes.

That exception is usually to do with downloading SpaPackStruct.xml.  Are you sure that you have manually tried copying that file to the HA root "config" directory?
Krs
Mark

Circling back to close the loop on this.  I got things working by:

using the fix for the async download of SpacPackStruct.xml
disconnecting the Mr. Steam Steamlinx unit from the network during the initial Hass discovery/configuration.

I can now restart Hass with both connected without issues.
I don't know if disconnecting the steamlinx unit during discovery/config was actually necessary.
At some point I should test with a fresh Hass instance that has no gecko config history. I'm closing this now and will reopen if needed.  Updates related to the steamlinx will be the related Geckolib issue - https://github.com/gazoodle/geckolib/issues/6


make compatible with react 15.5

solves #69

Ah, thanks for this!

Changes published as v1.0.0-rc.5.


Give Azure AD apps code the ability to add custom owners

PR Checklist

[x] I have added example(s) inside the [./examples/] folder
[ ] I have added the example(s) to the integration test list for normal (~30 minutes) or long runner >30 minutes
[x] I have checked the coding conventions as per the wiki
[x] I have checked to ensure there aren't other open Pull Requests for the same update/change?

Description

This small PR adds the ability to add custom owners to an app registration.
Does this introduce a breaking change

[ ] YES
[x] NO

Testing

Getting Null values are not allowed for this attribute value., so still need to add some logic to make a custom owner optional.


How to use full-text index for query?

I created a full-text index for a field, how should I use it in ent?
I didn't find the function of how to use full-text index in ent

Hey @kilogi and thanks for raising up this issue.
What database do you use? Can you please paste here the SQL statements/queries you'd like to execute with ent?

Hey @kilogi and thanks for raising up this issue.
What database do you use? Can you please paste here the SQL statements/queries you'd like to execute with ent?

Database is postgres.
sql statement like this “select title from posts where tsv_content @@ plainto_tsquery('hello world')”

The type of tsv_content field is tsvector and its index type is gin

How did u managed to create the index?

Hey! Just to follow up on this issue:
1. In case there's a need to create a FULLTEXT index, please use Atlas Annotation in your index definition:
func (User) Indexes() []ent.Index {
    return []ent.Index{
        index.Fields("c4").
            Annotations(entsql.IndexType("HASH")),
        // Enable FULLTEXT search on MySQL,
        // and GIN on PostgreSQL.
        index.Fields("c5").
            Annotations(
                entsql.IndexTypes(map[string]string{
                    dialect.MySQL:    "FULLTEXT",
                    dialect.Postgres: "GIN",
                }),
            ),
    }
}

2. In order to inject custom predicates or modifiers to queries, please see:

https://entgo.io/docs/predicates#custom-predicates
https://entgo.io/docs/feature-flags#custom-sql-modifiers

Closing, but I'm available in our Discord server for any question. Thanks 🙏


Add tests

Includes:

relevant testing for the main http handlers

Index
GetComments
CreateComment

TestMain that sets up and tears down a testing db.
Minor code fixes, as encountered when writing the tests.
Comments on exported values, for better godoc
Remove dot import from main.go

Fixes #56

@peteretelej cherry-picked your commits in https://github.com/adtac/commento/pull/101 :+1: (you're still the commit author, of course)


Basic multiline highlighting

This PR adds basic support for multiline highlighting:

The analysis stage of the conversion has remained roughly the same (i.e., most information was already available).
Most changes are in the transformation stage, which now cleanly separates the generation of "inner rules" (detect the internal content of a production) and "outer rules" (detect the external context of a production). See the new module ConversionUnit for more details in the comments.

All tests are updated accordingly as well (and some new ones were added).

Thank you, Toine, Davy, and Pieter for the comments 🙂


Modify SQL regex's to be case insensitive

Tried to use this, but with an upper-case AS in my SQL statement. Took me a bit to figure out why parsing was failing. This just modifies the SQL regex constants to be case-insensitive.

@MikeTheReader merged, thanks!


DATETIME field missing years with column_editable_list

I have set one of the fields, which is DATETIME type in the sqlite, into the column_editable_list, like this:
column_editable_list = ['finish_time']
However, where trying to edit it inline the year number only goes to 2015.
Any way to fix this?

Check in #1543 to see if that solves your problem

I have already check that.
I'm using the Flask-Admin 1.5.0, and the version of bootstrap.min.js file is 3.3.5, which I think is not the problem. But the year number still goes up to 2015.

https://github.com/flask-admin/flask-admin/commit/5fff53bc95466317cdff3c68a1b4d748666bf8b7
I have followed this commit above to make a change to the file: /flask_admin/static/admin/js/form.js
Everything goes well now!


HIVE-28259: Common table expression detection and rewrites using CBO

What changes were proposed in this pull request?

Add applyCteRewriting phase in CalcitePlanner for detecting and using CTEs; ensure rewrite logic is consistent with existing hive.optimize.cte.materialize.* properties.
Model CTEs as materialized views (MVs) and add utility method in HiveMaterializedViewUtils for mapping a CTE to a RelOptMaterialization.
Refactor core MV rewrite logic in CalcitePlanner to use during CTE rewrite and exploit CTEs in a cost-based manner.
Add HiveTableSpool operator to represent CTEs and handle them in the plan using new rules: TableScanToSpoolRule and RemoveUnusedCteRule.
Add TableScanToSpoolRule, and RemoveUnusedCteRule to add/remove spools from the plan.
Enhance/Enrich metadata handlers for handling the Spool operator.
Add AggregatedColumns metadata (and respective handler and metadata query), for controlling if a CTE is a "full aggregate" at the CBO (RelNode) level to ensure consistent behavior with hive.optimize.cte.materialize.full.aggregate.only property.
Add HiveSqlTypeUtil.containsSqlType for detecting and skipping the creation of CTEs with untyped nulls since they are not supported (HIVE-11217).
Add hive.optimize.cte.suggester.class and CommonTableExpressionSuggester interface to provide pluggable CTE detection logic. Given that CTE detection logic can range from basic tree traversal algorithms to complex workload analysis frameworks this part needs to be configurable since there is no one-size-fits-all implementation. The configuration property also allows proprietary algorithms to be integrated in HiveServer2 by implementing the necessary APIs and adding the jars in the classpath.
Add prototype implementation for CTE detection logic in CommonTableExpressionIdentitySuggester using CommonRelSubExprRegisterRule and CommonTableExpressionRegistry. Although the implementation is rather simple it can discover various interesting CTEs as demonstrated by the tests and can be indeed useful in a prod environment.
Map spool(s) to WITH clauses during the RelNode to AST conversion (ASTConverter, ASTBuilder, PlanModifierForASTConv) to exploit existing CTE materialization feature (HIVE-11752).
Modify (slighly) SemanticAnalyzer/CalcitePlanner to enable AST-based CTE materialization (getMetadata) post CBO run.

Why are the changes needed?

Open the road for cost-based CTE optimizations in Hive
Pluggable/Extensible CTE detection logic

Does this PR introduce any user-facing change?
By default no. After tuning the new/existing cte properties query plans may change affecting performance.
Is the change a dependency upgrade?
No
How was this patch tested?
Tests for:

CTE detection logic using the CommonTableExpressionPrintSuggester (TestTezTPCDS30TBPerfCliDriver)
demonstrating (end-to-end) the CTE feature (cte_cbo_rewrite_0.q)
verify coherence of CTE rewrite with hive.optimize.cte.materialize.full.aggregate.only (cte_mat_12.q)
spool JSON serialization (cte_cbo_plan_json.q)

Hey @asolimando , many thanks for reviewing the PR!! I was busy and didn't manage to reply to your comments. In the meantime the PR got a bit stale since master diverged. Would you mind if I rebase this PR cause merging with master may look messy?

Approving latest changes, there are 2 diff in qout files but it seems minor

The tests are green and the last small issues were resolved. If there are no objections I will merge this PR in ~24 or so.


Add a username and password to Blazegraph container

Currently the Blazegraph container required no authentication.

This should definitely result in an updated version number, as this seems to break tests when started from Windows (see the warning here https://github.com/cambridge-cares/TheWorldAvatar/tree/main/Agents/DerivationAgentPythonExample#test). Dockerised tests work fine within WSL, but copying credential files seems to cause permission issues when started from Windows.
So, just to keep in mind increasing version number (as you always do anyway). Just here for reference.

Please clarify what is meant by "the container". I believe this is about password-protecting certain URLs of the Tomcat webapp. Please also clarify which ones (all, just the web front-end, or also including the SPARQL endpoints), as this has the potential to break a lot of existing codes.

Please clarify what is meant by "the container". I believe this is about password-protecting certain URLs of the Tomcat webapp. Please also clarify which ones (all, just the web front-end, or also including the SPARQL endpoints), as this has the potential to break a lot of existing codes.

I have tried to pull the image docker.cmclinnovations.com/blazegraph:1.1.0-510-SNAPSHOT and can confirm that both web front-end and the SPARQL endpoint are protected. One can set up username and password when initialising RemoteStoreClient, and then the SPARQL operations via code will work fine. Otherwise, one will run into 401 Unauthorized error. I tested with PySparqlClient from pyderivationagent==1.4.1, so I suppose anything that uses RemoteStoreClient of the latest jps-base-lib should work.


Stop using contains for queries which use labels to match

I'm not sure if this is the perfect fix for #306 but I was able to get the new test to pass using this revised xpath.  That thing is so hard to follow though, I'm sure I might have missed something.  I'm opening this in the hopes to start a conversation about fixing this issue and I'm glad to help out.  If this does actually fix the issue then 🎉, but I'm betting there's more to it.
Also, what are your thoughts on making that xpaths easier to parse visually?  I spent some time breaking bits of it apart locally but didn't commit any of that here.  Perhaps I can offer some help there to lower the bar of entry for others, let me know.
Thanks for reviewing and thanks in advance for your feedback!

... and there is a check that labels have a for attribute.  Hmm...  I'll look into that.

@joelbyler Thanks for working on this! I'm sorry about the xpath stuff. It's gross. I originally intended on building a library to compose xpath queries but never made time for it. I haven't come up with a better solution yet but if you have thoughts I'd be interested in hearing them.

Hi guys 😄
is this still relevant? I would love to align this PR with the latest master if needed 😊
P.S: thanks a lot for this great library! 🙇


doc: update macOS_Catalina.md

The fixes suggested in macOS_Catalina.md are not working reliably in the latest versions of Catalina. @rrrix found that installing the Command Line Tools from dmg rather than xcode-select --install does work. This was confirmed by two other people, including myself.

Checklist

[x] documentation is changed or added

will defer to @cclauss on final approval of this

@cclauss Anything more you need from me on this? There have been another half dozen posts to 1927, so this is impacting a lot of folks right now.

Thanks for the contribution ❤


Update Hugo version to 0.101.0 for production builds

Summary
The version of Hugo used to build the docs in production is still using v0.92.1 and may be the cause of some rendering errors. This PR updates the version of Hugo to v0.101.0, so it matches the version used in GitHub actions.

Hello @neflyte,
Thanks for your pull request! A Core Committer will review your pull request soon. For code contributions, you can learn more about the review process here.


Bunny macro fails when queue has declared features

The macro fails because Bunny closes de connectiton when the queue has arguments declared. Currently the Bunny macro only supports x-max-priority but in our case we have queues with more features (such as x-dead-letter-exchange).
I would propose to the macro to accept a generic arguments hash instead of adding arguments one by one in the options hash.

I'm unfortunately not familiar with RabbitMQ/Bunny (this macro was contributed).
If you'd be willing to provide a PR I'd be happy to merge it in. It would be good if we could keep it backwards compatible. Perhaps introduce the arguments option that you mentioned and merge x-max-priority into it before passing it into queue(arguments:)? I think that way the existing API will continue to function as-is, while allowing for more flexibility around arguments.
i.e. something like this
options =
  if queues.last.is_a?(Hash)
    queues.pop
  else
    {}
  end

options[:arguments] ||= {}
options[:arguments][:"x-max-priority"] = options[:"x-max-priority"] if options.key?(:"x-max-priority")

# ....

channel.queue(queue_name, durable: options[:durable], arguments: options[:arguments])

https://github.com/hirefire/hirefire-resource/blob/master/lib/hirefire/macro/bunny.rb
What do you think?


Cannot import const enums

See https://github.com/cloudevents/sdk-javascript/issues/323
This is the code: https://github.com/cloudevents/sdk-javascript/blob/73f0becc2b8e4f10ae40e23b77e4161d9b5ff611/src/event/cloudevent.ts#L19-L22
It works with ts-jest, does not work with<EMAIL_ADDRESS>

see https://github.com/swc-project/swc/issues/940

Thanks, will close this then in favor of the issue linked.


move imager data to the clean bucket

Move this nasa data to the clean bucket
I have moved the following files to s3://na-stage.us-east-2.noaa-nasa-share/nasa/reanalysis/
as requested.
TMI,
AIRS
amsua-aqua
amsr2
gmi
amsre
avhrr_repro

@frolovsa I don't see any data in the bucket related to AIRS is that missing?
additionally, are these the folder names we want in the clean bucket? should it be just avhrr?

airs is in here aqua_disc/bufr/airs/ and it should go to airs/nasa/aqua_disc
aqua_disc/bufr/amsu/ should go to amsua/nasa/aqua/
AVHRR_repro ->avhrr/nasa/21cr_repro/
TRMM -> trmm/nasa/tmi
amsr2 -> amsr2/nasa/
amsre -> amsre/nasa/
gmi_v7->gmi/nasa/gmi_v7/
i hope this covers it


ItemSpacing on CollectionView fix

I don't know the reason for this constant that adds a margin between each item. Why not leave it at 0?
As it is right now, if we set ItemSpacing to 0 there is still a space generated by this constant, we need to set ItemSpacing = -2 (and change the code a bit to allow negative numbers) so that there is no spacing between items.
RecyclerViewItemVerticalMarginDp = 2 ItemSpacing = 0

RecyclerViewItemVerticalMarginDp = 0 ItemSpacing = 0

For reference, related issue :  #74

will be integrated in next version


#1466 Support custom class for action button and image

Fixes #1466
Custom classes for action button and image -

Developer's Certificate of Origin 1.1
   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

@nmgokhale You will need to update the pipeline schemas as well to support class_name in action_definition


Check existence of inet_pton for win32 in CMakeLists.txt (fixes #5019)

This is to fix #5019. We manually check the existence of inet_pton in CMakeLists.txt to avoid inet_pton being redefined in src/network/socket_wrapper.hpp.

Could you please add inet_pton existence check in configure.win too like it was stated in https://github.com/microsoft/LightGBM/issues/5019#issuecomment-1048289323?

@StrikerRUS Thanks for the reminder. I've added that in ff8c604. Could you please help to check?
BTW, do you have any idea why some actions of R packages with gcc and R4.1 hangs?

/gha run r-solaris

/gha run r-valgrind

/gha run r-valgrind

It looks like the valgrind build is being triggered by comments correctly
For example, https://github.com/microsoft/LightGBM/runs/6122965595?check_suite_focus=true

But is then failing prior to even building the R package.
https://github.com/microsoft/LightGBM/runs/6122967527?check_suite_focus=true
/usr/bin/git init /__w/LightGBM/LightGBM
  Initialized empty Git repository in /__w/LightGBM/LightGBM/.git/
  /usr/bin/git remote add origin https://github.com/microsoft/LightGBM
  Error: fatal: unsafe repository ('/__w/LightGBM/LightGBM' is owned by someone else)
  To add an exception for this directory, call:

  	git config --global --add safe.directory /__w/LightGBM/LightGBM
  Error: The process '/usr/bin/git' failed with exit code 1[28](https://github.com/microsoft/LightGBM/runs/6122967527?check_suite_focus=true#step:4:28)

I suspect that was just missed in #5152. Sorry about that!  Will put up a PR shortly.

/gha run r-solaris

/gha run r-valgrind

The r-solaris check fails with the following information
/usr/bin/ld: cannot find -lgfortran
collect2: error: ld returned 1 exit status
make[1]: *** [/usr/share/R/share/make/shlib.mk:10: Matrix.so] Error 1
make[1]: Leaving directory '/tmp/Rtmph9P7SX/R.INSTALL58396355090c/Matrix/src'
ERROR: compilation failed for package ‘Matrix’
* removing ‘/usr/local/lib/R/site-library/Matrix’

The r-solaris check fails

Still fails.
Maybe worth reporting to https://github.com/r-hub/rhub/issues?
I believe our actions cannot affect Matrix package compilatibility.
@jameslamb WDYT?

WDYT?

I agree! I see the following error at the top of the logs from the Solaris jobs, which I suspect is the root cause.
46#> Fatal server error:

[  47](https://builder.r-hub.io/status/lightgbm_<IP_ADDRESS>.tar.gz-4f49e421e97542b4876d80c696893832#L47)#> Could not create server lock file: /tmp/.X22-lock

I found one issue with a similar error, where maintainers point to "GitHub rate limiting" as a problem: https://github.com/r-hub/rhub/issues/492#issuecomment-966080730.
I also see the following in the logs:
Error: package or namespace load failed for ‘ps’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/export/home/XKCQ2Gb/R/00LOCK-ps/00new/ps/libs/ps.so':
  ld.so.1: R: fatal: relocation error: file /export/home/XKCQ2Gb/R/00LOCK-ps/00new/ps/libs/ps.so: symbol psll_memory_uss: referenced symbol not found

Anyway, I can try tonight submitting to R-hub from master. If that works, then maybe that means this was a temporary issue. If not, then I'll report it to {rhub} at that issues link.

Anyway, I can try tonight submitting to R-hub from master

I just built the package like this:
sh build-cran-package.sh

Then manually triggered Solaris builds like this:
package_tarball <- paste0("lightgbm_", readLines("VERSION.txt")[1], ".tar.gz")
rhub::check(
    path = package_tarball
    , email = "FILL_IN_EMAIL_HERE"
    , check_args = "--as-cran"
    , platform = c(
        "solaris-x86-patched"
        , "solaris-x86-patched-ods"
    )
    , env_vars = c(
        "R_COMPILE_AND_INSTALL_PACKAGES" = "always"
    )
)

With email set to the one linked in https://github.com/microsoft/LightGBM/pull/5159#issuecomment-1122807971.
Statuses should be sent there once the jobs complete.
Links:

https://builder.r-hub.io/status/lightgbm_<IP_ADDRESS>.tar.gz-49175e584edf47c9aa4e04e74ea750d9
https://builder.r-hub.io/status/lightgbm_<IP_ADDRESS>.tar.gz-de5f17b897844c598adddbd49782e707

Hasn't this meant that our token was overwritten?
https://github.com/microsoft/LightGBM/blob/0018206742544cfd0b2c1e156f177ab5f468a998/.ci/run_rhub_solaris_checks.R#L12-L15
https://github.com/microsoft/LightGBM/blob/0018206742544cfd0b2c1e156f177ab5f468a998/.ci/run_rhub_solaris_checks.R#L24-L27

I just get it! Matrix installation happens inside wch1/r-debug, not at RHub.
https://github.com/microsoft/LightGBM/blob/0018206742544cfd0b2c1e156f177ab5f468a998/.github/workflows/r_solaris.yml#L12
https://github.com/microsoft/LightGBM/blob/0018206742544cfd0b2c1e156f177ab5f468a998/.github/workflows/r_solaris.yml#L40-L42
https://github.com/microsoft/LightGBM/blob/0018206742544cfd0b2c1e156f177ab5f468a998/.ci/test_r_package_solaris.sh#L8-L10
Job fails even before submitting the LightGBM package to RHub.
https://github.com/microsoft/LightGBM/blob/0018206742544cfd0b2c1e156f177ab5f468a998/.ci/test_r_package_solaris.sh#L15
https://github.com/microsoft/LightGBM/blob/0018206742544cfd0b2c1e156f177ab5f468a998/.ci/run_rhub_solaris_checks.R#L37

Hasn't this meant that our token was overwritten?

It would surprise me if that was true. I assumed that when I was prompted for a new token, R-hub was issuing me an additional token for use with that email, and that you could have multiple associated with the same email address.
If that's not true and we find out that the token has been overwritten, then I apologize and I'll put up a PR to fix the comment-triggered jobs.

Job fails even before submitting the LightGBM package to RHub.

ooooh I see! I was wondering where @shiyu1994 saw failures related to {Matrix} when I didn't see that in the logs.
I guess then, based on https://github.com/microsoft/LightGBM/pull/5159#issuecomment-1121825932, we need to install gfortran into that job's environment prior to building the package.


ci: add e2e tests to publish-package action

Requirements:

add env variables for secrets to run e2e tests
run e2e tests as part of publish-package action
add flag to only deploy if e2e tests pass
add publish flag for publish-package action

Tests:

see this successful CI run for publish flag true (would publish but prints the flags both being true instead as test)  https://github.com/alchemyplatform/aa-sdk/actions/runs/5628745012/job/15252957921

see this successful CI run for publish flag false (skips deploy because flag is false) https://github.com/alchemyplatform/aa-sdk/actions/runs/5628721929/job/15252905766

see this correctly unsuccessful CI run for failed e2e tests (skips deploy because e2e failed even if publish flag is true) https://github.com/alchemyplatform/aa-sdk/actions/runs/5617640123

PR-Codex overview
This PR focuses on improving the test:e2e command and adding a workflow_dispatch event for manual triggering.
Detailed summary

Modified the test:e2e command to include the --concurrency=1 flag
Added a workflow_dispatch event with a publish input flag for manual triggering
Added environment variables for the E2E tests in the workflow
Updated the Publish using Lerna step to run only if the publish input flag is set to true

✨ Ask PR-Codex anything about this PR by commenting with /codex {your question}

Current dependencies on/for this PR:

main

PR #54

PR #55

PR #56

PR #57   👈

This comment was auto-generated by Graphite.

Current dependencies on/for this PR:

main

PR #54

PR #55

PR #56

PR #57   👈

This comment was auto-generated by Graphite.

can we add configuration params to this workflow? Did your test actually publish something?
EDIT: oh I see that it didn't. ok cool. Let's add the config params to this still so that we can leverage it in the future (probably pretty useful cuz we could theoretically run it manually on branches)

ah I should have clarified: I looked into config + conditional. Conditionals only work at the job level, not the job-step level. We'd need to refactor the publish action to have publish be a separate job after the install and build and test job. That cool?

@avasisht23 started a stack merge that includes this pull request via Graphite.

Graphite couldn't merge this pull request because a downstack PR #60 failed to merge.

@moldy530 started a stack merge that includes this pull request via Graphite.

@moldy530 merged this pull request with Graphite.


🐛 [firebase_database] MissingPluginException with certain unicode characters in filter

Bug report
Describe the bug
Whenever using the orderByChild(), startAt() and endAt() combination to filter data with some unicode characters such as \u{0000} and \u{FFFF}, a MissingPluginException is thrown. This happens when listening for data with onValue but not when executing the query with get().
The exception thrown is:
══╡ EXCEPTION CAUGHT BY SERVICES LIBRARY ╞══════════════════════════════════════════════════════════
The following MissingPluginException was thrown while activating platform stream on channel
streamers-[DEFAULT]-null-DatabaseEventType.value-[{type: orderBy, name: orderByChild, path:
activeSessionId}, {type: cursor, name: startAt, value: }, {type: cursor, name: endAt, value: }]#0:
MissingPluginException(No implementation found for method listen on channel
streamers-[DEFAULT]-null-DatabaseEventType.value-[{type: orderBy, name: orderByChild, path:
activeSessionId}, {type: cursor, name: startAt, value: }, {type: cursor, name: endAt, value: }]#0)

When the exception was thrown, this was the stack:
#0      MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:294:7)
<asynchronous suspension>
#1      EventChannel.receiveBroadcastStream.<anonymous closure> (package:flutter/src/services/platform_channel.dart:637:9)
<asynchronous suspension>
════════════════════════════════════════════════════════════════════════════════════════════════════

Steps to reproduce
Steps to reproduce the behavior:

Set up a query with orderByChild("someChild"), startAt("\u{0000}") and endAt("\u{FFFF}").
Listen to data using onValue and the created query, the exception will be thrown.

Expected behavior
The listener should not raise this exception and instead return data.
Sample project
https://github.com/juaoose/db_sample/blob/main/lib/main.dart#L47

Flutter doctor
Run flutter doctor and paste the output below:
Click To Expand
Doctor summary (to see all details, run flutter doctor -v):
[√] Flutter (Channel stable, 3.3.1, on Microsoft Windows [Version 10.0.22000.856], locale en-US)
[√] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
[√] Chrome - develop for the web
[X] Visual Studio - develop for Windows
    X Visual Studio not installed; this is necessary for Windows development.
      Download at https://visualstudio.microsoft.com/downloads/.
      Please install the "Desktop development with C++" workload, including all of its default components
[!] Android Studio (not installed)
[√] Connected device (4 available)
[√] HTTP Host Availability

! Doctor found issues in 2 categories.

Flutter dependencies
Run flutter pub deps -- --style=compact and paste the output below:
Click To Expand
Dart SDK 2.18.0
Flutter SDK 3.3.1
db_sample 1.0.0+1

dependencies:
- cupertino_icons 1.0.5
- firebase_core 1.22.0 [firebase_core_platform_interface firebase_core_web flutter meta]
- firebase_database 9.1.4 [firebase_core firebase_core_platform_interface firebase_database_platform_interface firebase_database_web flutter]
- flutter 0.0.0 [characters collection material_color_utilities meta vector_math sky_engine]

dev dependencies:
- flutter_lints 2.0.1 [lints]
- flutter_test 0.0.0 [flutter test_api path fake_async clock stack_trace vector_math async boolean_selector characters collection matcher material_color_utilities meta source_span stream_channel string_scanner term_glyph]

transitive dependencies:
- async 2.9.0 [collection meta]
- boolean_selector 2.1.0 [source_span string_scanner]
- characters 1.2.1
- clock 1.1.1
- collection 1.16.0
- fake_async 1.3.1 [clock collection]
- firebase_core_platform_interface 4.5.1 [collection flutter flutter_test meta plugin_platform_interface]
- firebase_core_web 1.7.2 [firebase_core_platform_interface flutter flutter_web_plugins js meta]
- firebase_database_platform_interface 0.2.2+4 [collection firebase_core flutter meta plugin_platform_interface]
- firebase_database_web 0.2.1+6 [firebase_core firebase_core_web firebase_database_platform_interface flutter flutter_web_plugins js]
- flutter_web_plugins 0.0.0 [flutter js characters collection material_color_utilities meta vector_math]
- js 0.6.4
- lints 2.0.0
- matcher 0.12.12 [stack_trace]
- material_color_utilities 0.1.5
- meta 1.8.0
- path 1.8.2
- plugin_platform_interface 2.1.3 [meta]
- sky_engine 0.0.99
- source_span 1.9.0 [collection path term_glyph]
- stack_trace 1.10.0 [path]
- stream_channel 2.1.0 [async]
- string_scanner 1.1.1 [source_span]
- term_glyph 1.2.1
- test_api 0.4.12 [async boolean_selector collection meta source_span stack_trace stream_channel string_scanner term_glyph matcher]
- vector_math 2.1.2

Thanks for the report. Using the code sample provided, I see same exception:
======== Exception caught by services library ======================================================
The following MissingPluginException was thrown while activating platform stream on channel counter-[DEFAULT]-null-DatabaseEventType.value-[{type: orderBy, name: orderByChild, path: messages}, {type: cursor, name: startAt, value: �}]#8:
MissingPluginException(No implementation found for method listen on channel counter-[DEFAULT]-null-DatabaseEventType.value-[{type: orderBy, name: orderByChild, path: messages}, {type: cursor, name: startAt, value: �}]#8)

When the exception was thrown, this was the stack:
#0      MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:313:7)
<asynchronous suspension>
#1      EventChannel.receiveBroadcastStream.<anonymous closure> (package:flutter/src/services/platform_channel.dart:662:9)
<asynchronous suspension>
====================================================================================================

Thanks for the report, I'm investigating

It seems linked to the Unicode character 0000 that is not supported by MethodChannel, the workaround would be to use any other character

Hey there! We did have an alternative solution to that character, the issue was mostly raised as this is a working query on other SDKs (such as node's). I guess having it documented for this SDK could eventually be helpful for someone else.
Thanks for looking into it :)


[py2py3] Migration at level src/python/A/B/C - start

Fixes #10055
Status
Ready
Related PRs

Next migration PR: #10103 (slice2, issue #10098  )

Description
Run futurize and some manual changes on the first batch of src/python/A/B/C.
Overview of the changes:

from builtins import str
from builtins import object
iterate over dictionary valus and items : viewitems, viewvalues

Overview based on file:

src/python/WMCore/Database/ResultSet.py

L39, r.keys():  r is not a dict, it is of type sqlalchemy.engine.result.RowProxy -> list(r) does not return a list of keys. Do not change that line with futurize!

Is it backward compatible (if not, which system it affects?)
It should be. Possible causes of errors:

src/python/WMCore/Services/UUIDLib.py returns a future.types.newstr.newstr instead of a py2 str. The test test/python/WMCore_t/Services_t/UUID_t.py has been updated accordingly

External dependencies / deployment changes
requires python-future
Open questions (to be addressed in follow up work)
src/python/WMCore/REST/Validation.py

is _check_str still needed? the docstring states "convert unicode to str (bytes) first to support cherrypy 3.2.2"

Since all unit tests passed, I will continue the migration in this PR in the coming days.

test this please

Besides a minor bug counting the number of pylint py3k events, how do you like it @mapellidario ? This does not have the futurize changes we discussed yesterday, I might handle that in a different pull request.

test this please

It is strange that we have a new unit test failing without any change, and it is not in this list . I tried rebasing this branch on top of master

Good that the unit test does not fail anymore. pylint --py3k report is very useful indeed! Thanks Alan!

test this please

I checked again the pylint --py3k report and it looks like it spotted a possible source of problems in py3 in WMCore/Agent/Daemonize/Create.py, involving the creation of file with file() instead of open().
I changed the labels and will have a better look a this tomorrow. @amaltaro  I will let you know when I fix this, then I will push the changes in a dedicated commit.

@amaltaro although there is no specefic unit test for Agent/Daemon/Create.py, the fact that no new unit started failing is promising (apart from that unstable one, which I assume is not a problem). What do you think?

Since in REST/Error.py we define message as an attribure of the RESTError class , here, pylint misleadingly suggests
    src/python/WMCore/REST/Error.py
        W1645, line 74 in RESTError.__str__: Exception.message removed in Python 3

But this is not a problem See some tests in py2 and py3 .

The warning from pylint py3k
src/python/WMCore/REST/Error.py
    W1645, line 74 in RESTError.__str__: Exception.message removed in Python 3

Anyway, it is a false positive because we manually added the .message attribute to the RESTError class:
class RESTError(Exception):
[...]
    http_code = None
    app_code = None
    message = None # this line makes the pylint --py3k warning a false positive
    info = None
    errid = None
    errobj = None
    trace = None

    def __init__(self, info = None, errobj = None, trace = None):
        self.errid = "%032x" % random.randrange(1 << 128)
        self.errobj = errobj
        self.info = info
        self.trace = trace
[...]

The warning from pylint py3k
src/python/WMCore/REST/Error.py
    W1645, line 74 in RESTError.__str__: Exception.message removed in Python 3

Anyway, it is a false positive because we manually added the .message attribute to the RESTError class:
class RESTError(Exception):
[...]
    http_code = None
    app_code = None
    message = None # this line makes the pylint --py3k warning a false positive
    info = None
    errid = None
    errobj = None
    trace = None

    def __init__(self, info = None, errobj = None, trace = None):
        self.errid = "%032x" % random.randrange(1 << 128)
        self.errobj = errobj
        self.info = info
        self.trace = trace
[...]

Squashed, thanks again!

Squashed, thanks again!

Thanks Dario.
It seems it failed to merge this PR, but I can actually see the corresponding commits under the commit history. Will wait a little to see if GH will fix itself, otherwise I'll close it.

Thanks Dario.
It seems it failed to merge this PR, but I can actually see the corresponding commits under the commit history. Will wait a little to see if GH will fix itself, otherwise I'll close it.

I also see the changes in master and they look good to me. This is weird.
Anyway, thanks for the review and for merging!

I also see the changes in master and they look good to me. This is weird.
Anyway, thanks for the review and for merging!

Thanks for double-checking it, Dario. Commits are in master, closing this PR.

Thanks for double-checking it, Dario. Commits are in master, closing this PR.


feat: add support for logs_body_key parameter on Opentelemetry output…

… plugin (https://github.com/fluent/fluent-operator/issues/1410)
What this PR does / why we need it:
Add support for logs_body_key  parameter on Opentelemetry output plugin.
Which issue(s) this PR fixes:
Fixes #1410
Does this PR introduced a user-facing change?

add support for `logs_body_key` parameter on Opentelemetry output plugin

@yilmazo Thank you！


Bug using node 14 on nvm.  Not changing to the corresponding compatible npm version for node.

Operating system and version:
Windows 10
nvm debug output:

nvm ls output:
Normal
    18.18.0
  * 14.21.3 (Currently using 64-bit executable)

How did you install nvm?

Downloaded .exe and installed it
What steps did you perform?
I installed node versions.
nvm install 14
nvm install 18
nvm use 14.21.3
npm
What happened?
When using node 14 (nvm use 14.X) it didnt install the NPM correctly for that node version.
i was getting errors using any "npm" command, even just "npm". I got the errors on the screenshot.
When using the nvm install 14, the console properly showed that the corresponding npm version had been installed with the node version.
What did you expect to happen?
When using nvm install 14, nvm also installing the corresponding compatible npm version with node 14
For the fix I had to install the correct npm manually.
npm install -g<EMAIL_ADDRESS>

You’re not using nvm, you’re using nvm-windows, an entirely different project.


getAll() with statuses

Maybe some dont want to get declined voucher's or a specific type of voucher

implemented with 95b7878a16329e339cd4cbd0bf146b1fa7d797a3


Make width 100%

It would be great, if you could change the width to 100%. Else the space is not used.

I like this idea as well, I've put up a sample implementation here https://github.com/ivanfilippov/PowerDNS-Admin/tree/width_fix for review

I somehow broke my branch and made an empty commit. I've now fixed that for review.

This is fixed in the new_ui branch. This is a new branch with a new template and I'm still testing it so I don't recommend it for daily use. Once merged into master this issue can be closed.

We can set the width 100% now with PR  #50


Cleanup issue with latest versions of brackets and plugin

Brackets version 1.10
plugin 2.0.5

Installed Ruby
Installed Sass
Created .brackets.json with following code.

{
"sass.enabled": false,
"path": {
"SASS/app.scss": {
"sass.enabled": true,
"sass.options": {
"outputDir": "../CSS/",
"includePaths": [],
"sourceComments": true,
"outputStyle": "nested"
},
"linting.collapsed": false
}
}
}

Created files in respective folder
Sass/app.scss
CSS/app.css

This is my scss file code.

Linked css file in index.html but in scss file i am getting cleanup warning. Please help. I am new to programming. I have just started learning coding.

I have just upgrade to bracket 1.10 and this issue pop up again (every time I upgrade bracket). Please Yelp................

some times when I downgrade bracket the error disappear, Although I hate this options it works.
2, Or I use Prepos to compile sass to css-----------------it works but the worst alternative.

This time I want to use the latest versions of bracket every time Adobe release

@nehamjadhav @Humphreto try this version, be sure to uninstall first. https://mega.nz/#!N0lDmB5Z!N32mfb_YTtt6facHH0vkoCPHWq9tWD--jHiKAUPyqQ8

@gverm That link no longers works. What happened to it?


Free data in the feasibility check.

We seem to have to free the data in the SCIP model whenever we call optimize!

should we merge #170 instead? The two are doing the same thing no?

Yes, I also added another call to free  Model. I'll merge the two later.

Closing this one as it coincides with #170


"No Target Architecture" user error from ntdef.h

I am trying to follow the "Write a Hello World Windows Driver (KMDF)" tutorial using the most recent version of VS, SDK, and WDK. I am targeting a machine with x64 architecture. When I try to build the project, I get a series of errors. I get 'identifier "PSLIST_HEADER" is undefined', along with a handful of other identifiers. I also get '#error directive: "No target architecture defined". The source of these errors are not in my driver.c file, but wdm.h and ntdef.h. I've been looking through the headers and I think I understand what is happening, but I don't know what to do to fix it. Here is the code from ntdef.h that is the source of my first user error:
#if defined(AMD64) || defined(X86) || defined(ARM64EC)
#define PROBE_ALIGNMENT( _s ) TYPE_ALIGNMENT( ULONG )
#elif defined(IA64) || defined(ARM) || defined(ARM64)
//
// TODO: WOWXX - Unblock ARM. Make all alignment checks ULONG for now.
//
#define PROBE_ALIGNMENT( _s ) TYPE_ALIGNMENT( ULONG )
#elif !defined(RC_INVOKED)
#error "No Target Architecture"
#endif
None of these 'if defined' cases allow for defining x64. This seems to be the root cause for all my errors: the header files are using these preprocessor directives that don't have a definition for x64. Note, I do not use #include <ntdef.h> That header is included via ntddk.h. I only use #include <ntddk.h> and #include <wdf.h>.
My path to ntddk.h is: C:\Program Files (x86)\Windows Kits\10\Include\10.0.22621.0\km
My path to wdf.h is C:\Program Files (x86)\Windows Kits\10\Include\wdf\kmdf\1.33
The tutorial does not discuss including a path to wdf.h, but VS could not find the wdf.h without adding a path using "Additional Include Directories."

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 81568a12-c39e-a18f-8f33-c54545f14bac
Version Independent ID: 73217b22-1795-b5d0-65cb-1984f38dc661
Content: Write a Hello World Windows Driver (KMDF) - Windows drivers
Content Source: windows-driver-docs-pr/gettingstarted/writing-a-very-small-kmdf--driver.md
Product: windows-hardware
Technology: gettingstarted
GitHub Login: @aviviano
Microsoft Alias: amvivian

Problem solved, I was missing Universal C Runtime from the VS Installer.


Problem with famous dependency

It looks like famous may have changed something, as they now have a src directory that contains subdirectories, but they don't have a main in package.json that reflects this. Bottom line is that running webpack-dev-server fails with these errors:
http://localhost:8080/webpack-dev-server/
webpack result is served from /dist/
content is served from /Users/tauren/github/AetherBoard/AetherBoard
Hash: 08607816b690cecadf78
Version: webpack 1.4.15
Time: 745ms
     Asset    Size  Chunks             Chunk Names
index.html     480          [emitted]
 bundle.js  615563       0  [emitted]  main
chunk    {0} bundle.js (main) 585453 [rendered]
    [0] ./client.js 868 {0} [built]
    [1] ./input.js 966 {0} [built]
    [2] ./invoke.js 334 {0} [built]
    [3] ./spline.js 809 {0} [built]
    [4] ./ui/index.js 709 {0} [built] [2 errors]
    [5] ./~/debug/browser.js 3464 {0} [built]
    [6] ./~/debug-stream/index.js 789 {0} [built]
    [7] ./ui/whiteboard.js 1204 {0} [built] [1 error]
    [8] ./~/graft/ws.js 1050 {0} [built]
    [9] ./~/graft/graft.js 2535 {0} [built]
   [10] ./~/famous-polyfills/index.js 104 {0} [built]
   [11] ./~/debug/debug.js 4096 {0} [built]
   [12] ./~/readable-stream/readable.js 451 {0} [built]
   [13] ./~/through2/through2.js 2140 {0} [built]
   [14] ./ui/index.html 55 {0} [built]
   [15] ./ui/sync.js 512 {0} [built] [5 errors]
   [16] ./~/lodash/dist/lodash.compat.js 244097 {0} [built]
   [17] ./ui/main.css 754 {0} [built]
   [18] ./~/css-loader!./ui/main.css 365 {0} [built] [1 error]
   [19] ./~/style-loader/addStyle.js 1128 {0} [built]
   [20] (webpack)/~/node-libs-browser/~/buffer/index.js 28521 {0} [built]
   [21] (webpack)/~/node-libs-browser/~/process/browser.js 2217 {0} [built]
   [22] ./~/graft/lib/server.js 1223 {0} [built]
   [23] ./~/graft/lib/client.js 1103 {0} [built]
   [24] ./~/graft/lib/deepMatch.js 344 {0} [built]
   [25] ./~/graft/lib/wrapMessage.js 488 {0} [built]
   [26] ./~/famous-polyfills/classList.js 3914 {0} [built]
   [27] ./~/famous-polyfills/functionPrototypeBind.js 780 {0} [built]
   [28] ./~/famous-polyfills/requestAnimationFrame.js 471 {0} [built]
   [29] ./~/debug-stream/~/split/index.js 1494 {0} [built]
   [30] ./~/readable-stream/lib/_stream_readable.js 27120 {0} [built]
   [31] ./~/readable-stream/lib/_stream_writable.js 10902 {0} [built]
   [32] ./~/readable-stream/lib/_stream_duplex.js 2811 {0} [built]
   [33] ./~/readable-stream/lib/_stream_transform.js 7396 {0} [built]
   [34] ./~/readable-stream/lib/_stream_passthrough.js 1727 {0} [built]
   [35] ./~/readable-stream/transform.js 55 {0} [built]
   [36] ./~/debug-stream/~/debug/browser.js 3277 {0} [built]
   [37] ./~/debug-stream/~/through2/through2.js 1719 {0} [built]
   [38] (webpack)/~/node-libs-browser/~/stream-browserify/index.js 3627 {0} [built]
   [39] (webpack)/~/node-libs-browser/~/util/util.js 15563 {0} [built]
   [40] ./~/graft/~/inherits/inherits_browser.js 672 {0} [built]
   [41] ./~/debug/~/ms/index.js 2025 {0} [built]
   [42] (webpack)/buildin/module.js 251 {0} [built]
   [43] ./~/graft/~/jschan/lib/jschan_browser.js 404 {0} [built]
   [44] ./~/through2/~/xtend/immutable.js 323 {0} [built]
   [45] ./~/debug-stream/~/debug/debug.js 4096 {0} [built]
   [46] (webpack)/~/node-libs-browser/~/buffer/~/is-array/index.js 470 {0} [built]
   [47] (webpack)/~/node-libs-browser/~/buffer/~/ieee754/index.js 2093 {0} [built]
   [48] (webpack)/~/node-libs-browser/~/string_decoder/index.js 7796 {0} [built]
   [49] (webpack)/~/node-libs-browser/~/events/events.js 8251 {0} [built]
   [50] ./~/debug-stream/~/split/~/through/index.js 2618 {0} [built]
   [51] ./~/readable-stream/~/isarray/index.js 120 {0} [built]
   [52] ./~/readable-stream/~/string_decoder/index.js 7796 {0} [built]
   [53] ./~/readable-stream/~/inherits/inherits_browser.js 672 {0} [built]
   [54] (webpack)/~/node-libs-browser/~/buffer/~/base64-js/lib/b64.js 3430 {0} [built]
   [55] (webpack)/~/node-libs-browser/~/util/support/isBufferBrowser.js 203 {0} [built]
   [56] ./~/readable-stream/~/core-util-is/lib/util.js 3040 {0} [built]
   [57] ./~/graft/~/jschan/lib/memory/session.js 1907 {0} [built]
   [58] ./~/graft/~/jschan/lib/stream/session.js 5752 {0} [built]
   [59] ./~/graft/~/jschan/lib/websocket/client.js 374 {0} [built]
   [60] ./~/debug-stream/~/through2/~/xtend/index.js 323 {0} [built]
   [61] (webpack)/~/node-libs-browser/~/readable-stream/readable.js 451 {0} [built]
   [62] (webpack)/~/node-libs-browser/~/readable-stream/writable.js 54 {0} [built]
   [63] (webpack)/~/node-libs-browser/~/readable-stream/duplex.js 52 {0} [built]
   [64] (webpack)/~/node-libs-browser/~/readable-stream/transform.js 55 {0} [built]
   [65] (webpack)/~/node-libs-browser/~/readable-stream/passthrough.js 57 {0} [built]
   [66] (webpack)/~/node-libs-browser/~/stream-browserify/~/inherits/inherits_browser.js 672 {0} [built]
   [67] (webpack)/~/node-libs-browser/~/util/~/inherits/inherits_browser.js 672 {0} [built]
   [68] ./~/debug-stream/~/debug/~/ms/index.js 2025 {0} [built]
   [69] ./~/graft/~/jschan/lib/memory/channel.js 2210 {0} [built]
   [70] ./~/graft/~/jschan/lib/stream/channels.js 2017 {0} [built]
   [71] ./~/graft/~/jschan/lib/encoder.js 4947 {0} [built]
   [72] (webpack)/~/node-libs-browser/~/readable-stream/lib/_stream_duplex.js 2811 {0} [built]
   [73] (webpack)/~/node-libs-browser/~/readable-stream/lib/_stream_writable.js 13069 {0} [built]
   [74] (webpack)/~/node-libs-browser/~/readable-stream/lib/_stream_transform.js 7350 {0} [built]
   [75] (webpack)/~/node-libs-browser/~/readable-stream/lib/_stream_readable.js 25959 {0} [built]
   [76] (webpack)/~/node-libs-browser/~/readable-stream/lib/_stream_passthrough.js 1727 {0} [built]
   [77] ./~/graft/~/jschan/~/reduplexer/index.js 3621 {0} [built]
   [78] ./~/graft/~/jschan/~/websocket-stream/index.js 1431 {0} [built]
   [79] ./~/graft/~/jschan/lib/channels.js 2471 {0} [built]
   [80] ./~/graft/~/jschan/~/async/lib/async.js 34598 {0} [built]
   [82] ./~/graft/~/jschan/~/msgpack5/index.js 13169 {0} [built]
   [83] (webpack)/~/node-libs-browser/~/readable-stream/~/inherits/inherits_browser.js 672 {0} [built]
   [84] (webpack)/~/node-libs-browser/~/readable-stream/~/core-util-is/lib/util.js 3040 {0} [built]
   [85] (webpack)/~/node-libs-browser/~/readable-stream/~/isarray/index.js 120 {0} [built]
   [86] ./~/graft/~/jschan/~/websocket-stream/~/duplexify/index.js 5331 {0} [built]
   [87] ./~/graft/~/jschan/~/ws/lib/browser.js 831 {0} [built]
   [88] ./~/graft/~/jschan/~/msgpack5/lib/streams.js 2227 {0} [built]
   [89] (webpack)/~/node-libs-browser/~/assert/assert.js 11706 {0} [built]
   [90] ./~/graft/~/jschan/~/msgpack5/~/bl/bl.js 4949 {0} [built]
   [91] ./~/graft/~/jschan/~/websocket-stream/~/duplexify/~/end-of-stream/index.js 2024 {0} [built]
   [92] ./~/graft/~/jschan/~/websocket-stream/~/duplexify/~/end-of-stream/~/once/once.js 417 {0} [built]
   [93] ./~/graft/~/jschan/~/websocket-stream/~/duplexify/~/end-of-stream/~/once/~/wrappy/wrappy.js 905 {0} [built]
     + 1 hidden modules

ERROR in ./ui/index.js
Module not found: Error: Cannot resolve module 'famous/core/Engine' in /Users/tauren/github/AetherBoard/AetherBoard/ui
 @ ./ui/index.js 9:13-42

ERROR in ./ui/index.js
Module not found: Error: Cannot resolve module 'famous/core/Modifier' in /Users/tauren/github/AetherBoard/AetherBoard/ui
 @ ./ui/index.js 10:17-48

ERROR in ./ui/whiteboard.js
Module not found: Error: Cannot resolve module 'famous/surfaces/CanvasSurface' in /Users/tauren/github/AetherBoard/AetherBoard/ui
 @ ./ui/whiteboard.js 1:13-53

ERROR in ./ui/sync.js
Module not found: Error: Cannot resolve module 'famous/core/Engine' in /Users/tauren/github/AetherBoard/AetherBoard/ui
 @ ./ui/sync.js 1:13-42

ERROR in ./ui/sync.js
Module not found: Error: Cannot resolve module 'famous/inputs/GenericSync' in /Users/tauren/github/AetherBoard/AetherBoard/ui
 @ ./ui/sync.js 3:18-54

ERROR in ./ui/sync.js
Module not found: Error: Cannot resolve module 'famous/inputs/MouseSync' in /Users/tauren/github/AetherBoard/AetherBoard/ui
 @ ./ui/sync.js 4:18-52

ERROR in ./ui/sync.js
Module not found: Error: Cannot resolve module 'famous/inputs/TouchSync' in /Users/tauren/github/AetherBoard/AetherBoard/ui
 @ ./ui/sync.js 5:18-52

ERROR in ./ui/sync.js
Module not found: Error: Cannot resolve module 'famous/inputs/ScrollSync' in /Users/tauren/github/AetherBoard/AetherBoard/ui
 @ ./ui/sync.js 6:18-53

ERROR in ./~/css-loader!./ui/main.css
Module not found: Error: Cannot resolve module 'famous/core/famous.css' in /Users/tauren/github/AetherBoard/AetherBoard/ui
 @ ./~/css-loader!./ui/main.css 2:1-113
webpack: bundle is now VALID.

I got it compiling again with some changes in webpack.config, because i was using a pre-release version of 0.3 before.
This also needs to be merged upstream, because there's a bug breaking webpack builds there too:
https://github.com/maxogden/websocket-stream/pull/59


add Action limits support

Closes #312
This PR is to add support for action limitations, including

update of struct definition
update of parser
unit test
validation

Adjusted comment text from "Close issue" to "Closes" to see if GitHub will automatically close the issue.  The word "issue" may have prevented this automation.

Please see comments made in issue #312

PR is updated to include emitting a WARNING if it's not changeable.
Please review again.


Fatal error: spawn cmd ENOENT

Please help me fixing this:
Running "connect:livereload" (connect) task
Started connect web server on http://localhost:9000
spawn called
{ '0': 'cmd',
'1': [ '/c', 'start', 'http://localhost:9000' ],
'2': { stdio: 'ignore' } }
Fatal error: spawn cmd ENOENT

Support questions are better asked on StackOverflow:
http://stackoverflow.com/questions/tagged/gruntjs
Paste in the link to the question here so others can follow ;)