Test cases written in moto using Latest version of boto3 fails

Seeing this issue is not resolved https://github.com/spulec/moto/issues/1793
I am using
boto = 2.49.0
boto3 = 1.14.57
with moto 1.3.16 I receive the following error
botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the Publish operation: No account found for the given parameters

../../.local/share/virtualenvs/grs-wcclaims-note-ingestion-service-enGYxXYk/lib/python3.8/site-packages/botocore/client.py:656: ClientError

with moto 1.3.10 there is no issues
    @mock_sns
    def test_publish_sns_message_success(self):
        sns_client_mockd = get_sns_client()
        sns_client_mockd.create_topic(Name=get_sns_topic_name())
        response = sns_client_mockd.list_topics()
        topic_arn = response["Topics"][0]["TopicArn"]

        os.environ["TOPIC_ARN"] = topic_arn

        edm_response = get_edm_response()
        output = publish_message_to_topic(edm_response)

def publish_message_to_topic(edm_response):
    message_to_publish = json.loads(edm_response)

    response = client.publish(
        TargetArn=os.environ["TOPIC_ARN"],
        Message=json.dumps({"default": message_to_publish["data"]}),
        MessageStructure="json",
    )

I confirm that the @mock_dynamodb2 decorator doesn't work with moto v1.3.16 (boto-2.49.0, boto3-1.14.60, botocore-1.12.253).

Hi @samsonych, are you getting the same error message?
This might fix your issue:
https://github.com/spulec/moto/blob/master/README.md#very-important----recommended-usage

Going to close this now, as it should be fixed by following the above guidelines.
Let me know if you run into any other issues.


Training VN_DGCNN_partseg on own dataset

Hi there!
Goal:
I'm trying to figure out how to train the VN_DGCNN on my own dataset for partial segmentation. I have just one folder (instead of 16 like in the ShapeNet dataset), i.e. one object I'd like to train this model on.
Problem:
I assume I just need to alter the dataloader script (which I did; and the train-file naturally), but it's returning an error still. If I interpret the error correctly, it expects weights for 16 objects still?
If anyone can point me into a certain direction, that'd be much appreciated!
Error:
Traceback (most recent call last):   File "/home/~/VNN/train_partseg_US.py", line 320, in <module>     main(args)   File "/home/~/VNN/train_partseg_US.py", line 204, in main     seg_pred, trans_feat = classifier(points, to_categorical(label, num_classes))   File "/home/~/.conda/envs/VNN/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl     return forward_call(*input, **kwargs)   File "/home/~/VNN/models/vn_dgcnn_partseg.py", line 86, in forward     l = self.conv7(l)   File "/home/~/.conda/envs/VNN/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl     return forward_call(*input, **kwargs)   File "/home/~/.conda/envs/VNN/lib/python3.9/site-packages/torch/nn/modules/container.py", line 141, in forward     input = module(input)   File "/home/~/.conda/envs/VNN/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl     return forward_call(*input, **kwargs)   File "/home/~/.conda/envs/VNN/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 302, in forward     return self._conv_forward(input, self.weight, self.bias)   File "/home/~/.conda/envs/VNN/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 298, in _conv_forward     return F.conv1d(input, weight, bias, self.stride, RuntimeError: Given groups=1, weight of size [64, 16, 1], expected input[16, 1, 1] to have 16 channels, but got 1 channels instead

Alright, so here's the story in short:

I fixed it.
I got the solution right away, weeks ago, but I adjusted the wrong model (vn_pointnet) by putting the number of classes (1 in this case) in the conv7 layer. So naturally nothing happened at that point. Derp.
This was an unnecessary post, but hopefully still useful for amateurs like myself.

Hi there!
I haved trained the VN_DGCNN model on my own dataset with 4096 point_num per PointCloud,while it took up a lot of video memory.When I set the batchsize to 22 with 4096 pointnum per frame, it almost took up totally 60G video memory  on three GeForce RTX 3090(with 24G per card).
So I would like to ask if you also have the problem of using too much memory during your training.
Thank you!


Death Knight HelpIDs Broken

The helpIDs don't work and break the frames.

Bad spell IDs?

The spell ids should be fine but they don't work.


Update SLF4J to 2+ (Fix for #140)

Right now we are using deprecated slf4j and deprecated slf4j-timber.
I updated slf4j-timber to support up to 2+ and publish it via Maven Central:
https://github.com/LionZXY/slf4j2-timber
Fix for https://github.com/NordicSemiconductor/Kotlin-BLE-Library/issues/140

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

That is great! Thank you!

I let myself to make a PR to your library: https://github.com/LionZXY/slf4j2-timber/pull/2
Anyway, looks great, I'll test and see how it rolls.

Fixed!


Add marginalia file privilege faces

Now that https://github.com/minad/marginalia/pull/91 has been merged, Marginalia has new faces to do exactly the same thing a diredfl currently does with file attributes. I.e. this:

So, here's a PR that just copies the diredfl face information, just for the incoming Marginalia faces.

Hello @tecosaur!  This relates to what we discussed earlier at the marginalia repo: https://github.com/minad/marginalia/pull/91#issuecomment-886821035
I think there is one small change that could be done which would (i) not change the character of the theme, and (ii) ensure greater consistency between diredfl and marginalia.  Please note that I am not familiar with the theme and thus cannot suggest more thoroughgoing reforms.
The diff (in case you want to send a patch):
@@ -439,7 +439,7 @@
     (diredfl-ignored-file-name      :foreground comments)
     (diredfl-link-priv              :foreground violet)
     (diredfl-no-priv                :foreground fg)
-    (diredfl-number                 :foreground orange)
+    (diredfl-number                 :foreground violet)
     (diredfl-other-priv             :foreground magenta)
     (diredfl-rare-priv              :foreground fg)
     (diredfl-read-priv              :foreground yellow)

Before VS After:

I would not do it the other way around (add orange to marginalia-size or marginalia-number) because I think that draws more attention to the number than what it really needs.

Thanks @protesilaos! I'll have a look at that :slightly_smiling_face:


Helm addons

What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #2398
What type of PR is this?
/kind design
Special notes for your reviewer:
helm libs imports are provided just for demonstration purposes, and will be deleted prior merge
Does this PR introduce a user-facing change? Then add your Release Note here:

helm addons proposal

Documentation:

NONE

/assign @xmudrii
PTAL


[ICD] Android chip-tool can buffer IM message when ICD device is not active.

In android chip-tool, under "ExperimentalLITInteraction" UI, we would automatically create the wildcard subscription, via subscription, chip-tool can know whether the LIT becomes inactive and further auto-queue via OnResubscriptionNeeded+ CHIP_ERROR_LIT_SUBSCRIBE_INACTIVE_TIMEOUT, which may be the only callback we can rely on regarding whether the device becomes inactive, then if check-in and report comes, we flush queue.  If user really wanna interact with LIT immediately, they can click "Force Interaction UI" under "ExperimentalLITInteraction", which instructs user to manually wake-up device, once check-in is received,, it is expected to automatically send StayActive Command with specified large value.

not to implement this queue per tiger team meeting.


Additional list functions

[ ] all
[ ] any
[ ] at
[ ] concat
[ ] index_map
[ ] intersperse
[ ] sort
[ ] unique
[ ] zip

Hi! This looks like a good first issue to tackle. I'd like to work on it :)

Wonderful! If you have any questions ask them here or in the irc channel. :)
Happy to accept multiple PRs or one big one, whatever works for you.

Is the plan to have a list module to encapsulate these functions?

There is one already! See gleam_stdlib/src/list.gleam

@lpil
Is concat meant to return a flattened list? [[1],[2]] -> [1,2]
Or a string? [1,2] -> "12"

@lpil  Any takes a predicate function and returns true if all the elements in the list return True. - Isn't this all?

Sorry, typo on my part. That is indeed all! Any is if at least one returns True for that predicate. contains doesn't take a predicate, instead it just checks for equality against each element.


Test case for a select statement with prepared statement with parameters

See #443 for background.
This is a non-failing test case that tests PreparedStatement.setTimestamp.
To fix #443, the data will be changed to .999 (from .003).

@gs-rezaem,
Thanks for your contribution.
To ensure that the project team has proper rights to use your work, please complete the Contribution License Agreement at https://cla.microsoft.com.
It will cover your contributions to all Microsoft-managed open source projects.
Thanks,
Microsoft Pull Request Bot

@gs-rezaem, thanks for signing the contribution license agreement. We will now validate the agreement and then the pull request.
Thanks, Microsoft Pull Request Bot

Looking at the failure.

I need a little help here. The test passes locally. I thought it has to do with timezones, so I did everything as UTC, but it doesn't seem to help.

Hi @gs-rezaem. I tried out the test locally, and it failed for me too - until I changed the col1 to col2 on line 70. I believe this error is caused by the issue we're trying to fix (the rounding error), and it's passing for you locally because you have the fix already (it fails for me, so that's what I imagine).
Please let me know if you run into anything else. Thanks for your contribution 👍

Codecov Report

Merging #492 into master will decrease coverage by 0.01%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master     #492      +/-   ##
============================================
- Coverage      46.3%   46.28%   -0.02%
+ Complexity     2209     2205       -4
============================================
  Files           108      108
  Lines         25231    25231
  Branches       4173     4173
============================================
- Hits          11682    11678       -4
+ Misses        11537    11535       -2
- Partials       2012     2018       +6

Flag
Coverage Δ
Complexity Δ

#JDBC41
46.14% <ø> (+0.06%)
2197 <ø> (ø)
:arrow_down:

#JDBC42
46.17% <ø> (+0.02%)
2201 <ø> (ø)
:arrow_down:

Impacted Files
Coverage Δ
Complexity Δ

...om/microsoft/sqlserver/jdbc/ReaderInputStream.java
41.57% <0%> (-3.38%)
14% <0%> (-2%)

...a/com/microsoft/sqlserver/jdbc/PLPInputStream.java
53.01% <0%> (-1.21%)
29% <0%> (-1%)

...c/main/java/com/microsoft/sqlserver/jdbc/Util.java
59.17% <0%> (-0.44%)
88% <0%> (-1%)

...ncurrentlinkedhashmap/ConcurrentLinkedHashMap.java
42.2% <0%> (-0.22%)
46% <0%> (ø)

...in/java/com/microsoft/sqlserver/jdbc/IOBuffer.java
53.38% <0%> (-0.07%)
0% <0%> (ø)

...om/microsoft/sqlserver/jdbc/SQLServerBulkCopy.java
52.64% <0%> (+0.12%)
239% <0%> (ø)
:arrow_down:

...m/microsoft/sqlserver/jdbc/SQLServerResultSet.java
33.14% <0%> (+0.16%)
244% <0%> (ø)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b9cdef4...3fea5f5. Read the comment docs.

Codecov Report

Merging #492 into master will decrease coverage by 0.01%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master     #492      +/-   ##
============================================
- Coverage      46.3%   46.28%   -0.02%
+ Complexity     2209     2205       -4
============================================
  Files           108      108
  Lines         25231    25231
  Branches       4173     4173
============================================
- Hits          11682    11678       -4
+ Misses        11537    11535       -2
- Partials       2012     2018       +6

Flag
Coverage Δ
Complexity Δ

#JDBC41
46.14% <ø> (+0.06%)
2197 <ø> (ø)
:arrow_down:

#JDBC42
46.17% <ø> (+0.02%)
2201 <ø> (ø)
:arrow_down:

Impacted Files
Coverage Δ
Complexity Δ

...om/microsoft/sqlserver/jdbc/ReaderInputStream.java
41.57% <0%> (-3.38%)
14% <0%> (-2%)

...a/com/microsoft/sqlserver/jdbc/PLPInputStream.java
53.01% <0%> (-1.21%)
29% <0%> (-1%)

...c/main/java/com/microsoft/sqlserver/jdbc/Util.java
59.17% <0%> (-0.44%)
88% <0%> (-1%)

...ncurrentlinkedhashmap/ConcurrentLinkedHashMap.java
42.2% <0%> (-0.22%)
46% <0%> (ø)

...in/java/com/microsoft/sqlserver/jdbc/IOBuffer.java
53.38% <0%> (-0.07%)
0% <0%> (ø)

...om/microsoft/sqlserver/jdbc/SQLServerBulkCopy.java
52.64% <0%> (+0.12%)
239% <0%> (ø)
:arrow_down:

...m/microsoft/sqlserver/jdbc/SQLServerResultSet.java
33.14% <0%> (+0.16%)
244% <0%> (ø)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b9cdef4...3fea5f5. Read the comment docs.

So I changed it to 000 and it passes here. I'd chosen 003 because it doesn't have a rounding issue (the db can store 003 as a datetime). That's quite concerning, because the only difference I can think of is that I'm on an old version of SQL Server (10), so this problem may have become worse?

Work in progress... looking at the tests

need a little help here. I'm trying to run the encrypted tests and I'm getting this error:
org.opentest4j.TestAbortedException: Assumption failed: Aborting test case since no java key store and alias name exists!

	at org.junit.jupiter.api.Assumptions.throwTestAbortedException(Assumptions.java:246)
	at org.junit.jupiter.api.Assumptions.assumeTrue(Assumptions.java:119)
	at org.junit.jupiter.api.Assumptions.assumeTrue(Assumptions.java:104)
	at com.microsoft.sqlserver.jdbc.AlwaysEncrypted.AESetup.readFromFile(AESetup.java:147)
	at com.microsoft.sqlserver.jdbc.AlwaysEncrypted.AESetup.setUpConnection(AESetup.java:98)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)

Hi @gs-rezaem . You need to create the certificates in order to be able to run the Always Encrypted tests. This wiki page will guide you in setting up the environment.  Thanks.

Hi @gs-rezaem, issue #493 has been fixed, so we can work on this issue again. However, the PR's been open for a while and there's some conflicts. Would you like to resolve these yourself, or would you like for me to take over this PR and continue where you left off? Thanks.

@peterbae As I no longer have easy access to MS SQL, please go ahead and take over. Thanks.

Hi @gs-rezaem,
Thank you again for the test case. With #1687, the issue you originally were seeing can be avoided using the datetimeParameterType set to dateTime. Please let us know if this solution (available in 12.2+ works for you). If not, we can take another look at the "legacyDateTime" option you have here and whether its something we can incorporate. Closing, but will reopen if necessary.


Update gorm to v2

該当イシュー
外部仕様の変更点
none
内部仕様の変更点

Update gorm version to v2
dbinit コマンドを追加した（テーブル作成だるいので）

動作を保証するためのテストケースの詳細

Codecov Report

Merging #37 (618baf8) into master (5970974) will increase coverage by 2.72%.
The diff coverage is 63.41%.

@@            Coverage Diff             @@
##           master      #37      +/-   ##
==========================================
+ Coverage   66.66%   69.38%   +2.72%
==========================================
  Files           2        3       +1
  Lines          48       49       +1
==========================================
+ Hits           32       34       +2
- Misses         11       13       +2
+ Partials        5        2       -3

Flag
Coverage Δ

unittests
69.38% <63.41%> (+2.72%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

src/invitation/usecase.go
40.00% <40.00%> (ø)

pkg/infra/dao/error.go
60.00% <60.00%> (ø)

pkg/infra/dao/invitation.go
86.20% <80.95%> (+19.54%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d1abe7a...618baf8. Read the comment docs.

Codecov Report

Merging #37 (618baf8) into master (5970974) will increase coverage by 2.72%.
The diff coverage is 63.41%.

@@            Coverage Diff             @@
##           master      #37      +/-   ##
==========================================
+ Coverage   66.66%   69.38%   +2.72%
==========================================
  Files           2        3       +1
  Lines          48       49       +1
==========================================
+ Hits           32       34       +2
- Misses         11       13       +2
+ Partials        5        2       -3

Flag
Coverage Δ

unittests
69.38% <63.41%> (+2.72%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

src/invitation/usecase.go
40.00% <40.00%> (ø)

pkg/infra/dao/error.go
60.00% <60.00%> (ø)

pkg/infra/dao/invitation.go
86.20% <80.95%> (+19.54%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d1abe7a...618baf8. Read the comment docs.


Not able to conect to http://:5000, SAS Server is not starting

Please I need some help with the issue below.
I am trying to run the test harness on the Ubuntu, need some help in the below issue,
    <pemFilePath>certificates/sas_certificate.pem</pemFilePath>
    <keyFilePath>certificates/sas_private_key.key</keyFilePath>
    <caCerts>certificates/rootCA_certificate.pem</caCerts>
    <cpiCert>certificates/cpi_certificate.pem</cpiCert>

While generating the above certificates, I have few questions.
Also want to know the values for these if I'm running locally.
Country Name (2 letter code) [XX]:US
Organization Name (eg, company) [Default Company Ltd]:Airspan Networks
Organizational Unit Name (eg, section) []:RSA SAS Provider CA9001
Common Name (eg, your name or your server's hostname) []:WInnForum RSA SAS Provider CA
Also i have query regarding the above params to be given.
After generating all the certificates as per the how to generate file, below is the issue.
$ python2.7 StartOfProject.py
WINNF TEST HARNESS RELEASE: <IP_ADDRESS> - 2018-November-13
Please enter a csv file path that include the test you request to run (pay attention insert the csv file without the csv suffix)
, type quit and press ctrl+c to close
WINNF.FT.C.REG.1
would you like to add the test to specific folder ? (select yes or no)
no
the selected test from the user : WINNF.FT.C.REG.1 is starting now
Starting Server
When i run now it dosent give any error but i'm not able to reach the https://:5000
$ wget https://:5000
--2022-03-14 13:02:28--  https://:5000/
Connecting to :5000... connected.
OpenSSL: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
Unable to establish SSL connection.
please do the needful.

If your problem is not solved, you can follow my practice:
When generating a self-signed certificate,
replace all sub-cas with root CA
and change the common name to the IP address +Port of your SAS-Server


Feature/vdx 173 schema gen

FYI
I was having build problems with SSI-SDK. Locally it worked but not in the CI. When I tried "tsc --build --clean && tsc --build" as build script it started working.
I asked Chat GPT why:
Sometimes the build process can be more complex, and there may be dependencies between projects or source files that could result in an incorrect build order. In these cases, running tsc --build by itself may not be sufficient to ensure that your project is built correctly.
By running tsc --build --clean && tsc --build, you're forcing TypeScript to recompile all the files in your project from scratch, including any dependencies that may have been missed in the previous build. This can help to avoid any issues that might arise from a mismatched build order or missed dependencies.

Still not building consistently

Codecov Report

:exclamation: No coverage uploaded for pull request base (feature/VDX-165@b12912b). Click here to learn what that means.
The diff coverage is n/a.

:exclamation: Current head 5f1cff6 differs from pull request most recent head 9f18a4d. Consider uploading reports for the commit 9f18a4d to get more accurate results

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@                Coverage Diff                 @@
##             feature/VDX-165      #85   +/-   ##
==================================================
  Coverage                   ?   44.37%
==================================================
  Files                      ?       67
  Lines                      ?     2578
  Branches                   ?      575
==================================================
  Hits                       ?     1144
  Misses                     ?     1434
  Partials                   ?        0

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


fix: landing page image css fix

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] All the files I changed are in the same world language (for example: only English changes, or only Chinese changes, etc.)
[x] My changes do not use shortened URLs or affiliate links.

Closes #35549

Thank you for the fix, that was spot on. @alan-nguy

@alan-nguy @ahmadabdolsaheb You amended sponsor.css file. Did you check how the sponsors' page looks like after the fix?


Merging DoubleMachineLearning into MultiplyRobust

@houssamzenati
Inside the med_bench_prototype code, you implemented MultiplyRobust' fit function like this :
def fit(self, t, m, x, y):
        """Fits nuisance parameters to data
        """
        # bucketize if needed
        t, m, x, y = self.resize(t, m, x, y)

        # fit nuisance functions
        self.fit_score_nuisances(t, m, x, y)

        if self._ratio == 'density':
            self.fit_treatment_propensity_x_nuisance(t, x)
            self.fit_mediator_nuisance(t, m, x)

        elif self._ratio == 'propensities':
            self.fit_treatment_propensity_x_nuisance(t, x)
            self.fit_treatment_propensity_xm_nuisance(t, m, x)

        else:
            raise NotImplementedError

        if self._procedure == 'nesting':
            self.fit_cross_conditional_mean_outcome_nuisance(t, m, x, y)

        elif self._procedure == 'discrete':

            if not is_array_integer(m):
                self._bucketizer = KMeans(n_clusters=10, random_state=self.rng,
                                          n_init="auto").fit(m)
                m = self._bucketizer.predict(m)
            self.fit_mediator_nuisance(t, m, x)
            self.fit_cross_conditional_mean_outcome_nuisance_discrete(t, m, x, y)

        else:
            raise NotImplementedError

        self._fitted = True

        if self.verbose:
            print(f"Nuisance models fitted")

We can see that there's two discrimination variables : ratio and procedure
From my understanding about our discussion today, with some questions in the middle :

DML estimator is equivalent to the ratio='propensities' scenario of the MultiplyRobust estimator, where m is continuous (and multidimensional ?).
procedure variable enables to specify, in the scenario where m is continuous, if we want to "discretize" m, in order to make the estimation faster ?
We want to get rid of this discrete scenario, and therefore also get rid of the procedure variable
Additionally, we want to get rid of the ratio variable and detect automatically if we are in the density (m is discrete) or propensities (m is continuous or discrete) scenario ?!

fyi @judithabk6

I may have missed some of the previous steps, but I would add the logic to infer such parameters automatically after the current refactoring.  It's an optimization, not really a must-have imho.

@bthirion one of the first step included some legacy code from @houssamzenati, so we are rather reverting back to the current state
not a must-have but not so complicated I think?

@brash6 yes I agree with what you wrote, it is indeed what we said yesterday @bthirion we are indeed trying to implement automatic inference of such variants, the only parameter that will be let is ratio='propensities' or 'density' for the mediator m when it is only discrete only. Would you like to also enforce a single option in that case? we thought with judith that in that case it was beneficial to let the choice.

@houssamzenati It's clear on my side, so :

We remove the discrete scenario (and therefore the _procedure variable)
We keep the _ratio variable to enable the choice between propensities and density when m is discrete
Finally, do we remove DML in this PR (from my understanding, if DML is just a subcase of MR, ce can just remove it in this PR after the two first implementations are done ? @bthirion @houssamzenati @judithabk6

I agree with the 3 points

yes you can remove DML in the PR after the points are treated and also when cross fitting is implemented (is back actually) in the code.


added yaml files for hunting queries and analytic rules

Required items, please complete
Change(s):

Added hunting queries and analytic rules for CloudAppEvents

Reason for Change(s):

New additions for CloudAppEvents table

Version Updated:

Required only for Detections/Analytic Rule templates

Testing Completed:
Checked that the validations are passing and have addressed any issues that are present:

Hi @jamiehuang00,
Please work on above comments.
Thanks

Hi @jamiehuang00,
Please work on above comment.
Thanks

@shainw Could you please take a quick look as well before we merge ? Thanks.


Use propper icons instead of emojis

Links to look through:

Bootstrap Icons
React Bootstrap Icons Package

Fixed with commit 2e7efa093038a435c17df22476292a146d5fe760


how to use queydsl in  elasticsearch java project?

https://github.com/querydsl/querydsl-contrib
Querydsl Elasticsearch Not updated ?Not maintained?
how to use queydsl in  elasticsearch java project?querydsl new version Support？

I'm sorry to say that Elasticsearch is a community project, and as such is not actively maintained by the team (or anyone else for that matter).
Leaving this open for a little while in case someone has input on the issue, but I wouldn't hold my breath.


Timeline View of Student Progress

Needs a small bit of cleanup. I mainly want to make the main timeline generation loop cleaner.
Example from local view:

Did you run ./manage.py resetdb? (Otherwise the old db won't have messages with analytics)

^ Looks like a bad merge for tests.

Nothing blocking though, feel free to merge

There's no rush to merge so I'll try to address them. I'll at least fix the tests.

Going to merge & deploy after tests run.


[New Port Request] ApprovalTests.cpp

Library name: ApprovalTests.cpp
Library description: Approval Tests allow you to verify a chunk of output (such as a file) in one operation as opposed to writing test assertions for each element.
Source repository URL: https://github.com/approvals/ApprovalTests.cpp
Project homepage (if different from the source repository):
Anything else that is useful to know when adding (such as optional features the library may have that should be included):

It's released as a single-header-only file
Dependency on unit test frameworks

ApprovalTests.cpp is effectively a convenience layer on top of other unit test frameworks.
As of January 2021, the supported test frameworks are GoogleTest, Catch2, CppUTest, doctest, Boost.Test, [Boost].UT
So I presume that somehow the vcpkg port will need to give users a way to specify which one of those libraries to add a dependency on
See the option names used for that in Conan

If you wish to add this port in the future, you can reopen this issue, but we're closing it for now.


Initial canright_composed proof using sbox correctness axioms

This introduces the circuit_equiv inductive type to allow proof automation on Circuit structure. canright_composed is proved correct to a specification under assumption of correctness of sub components.
Co-authored-by: jadephilipoom<EMAIL_ADDRESS>

For GitHub record: This is a re-merge of #215


Create Geneva branch and correct the compose file location

The Geneva docker-compose file location needs to be corrected due to https://github.com/edgexfoundry/developer-scripts/pull/288

The branch should be created from tag v1.2.0

fixed by #114


Shortcuts do not work with iterm2 and external keyboard

Checklist

[x] I have searched for existing issues
[x] I have looked through the wiki
[x] I have updated MonitorControl to the latest version

Describe the bug
When using iterm2 and monitor control the shortcuts to change brightness are not detected by monitor control.
To Reproduce
Steps to reproduce the behavior:

Install iterm2 and monitor control
Connct an external keyboard and monitor
Open iterm2 and give it focus
Press the f14 / f15 shortcuts to change brightness

Expected behavior
Brightness changes
Actual
Brightness does not change
Additional context
This might be an iterm2 only issue, but I wanted to see if anyone else has hit this or not.
Environment Information (please complete the following information):

macOS version: 10.15.7
MonitorControl version: 2.1.0
Monitor(s): viewsonic vp2768-4k

Possibly a duplicate of #282? Could be related to the secure text entry setting.

Yep! Thank you! Not sure why I didn't find that in my search...


Add logging for start and finish asset transfer

Category

[ ] Bug
[X] Enhancement

Expected or Desired Behavior
When assets are transferring, esp those of reasonable size, there is a perceivable pause in the logging output especially when in console logging mode - thus giving the impression the app is stuck, when its just uploading the data.
Add more logging points in asset transfer.

Makes sense. Do you want to include this in the November release or more thinking of the December release? For November the cut off date would be this week Thursday.

Yeap, ill add something before then - thanks for the deadline note.

Your change has been merged @pkbullock , thanks!


Support unretained Variables in _check_grad_type

Close #5608.

Jenkins, test this please.

Jenkins CI test (for commit 00906243b5cafc00a827a300e41eca7385e11219, target branch master) failed with status FAILURE.
(For contributors, please wait until the reviewer confirms the details of the error.)

Merge #5727 first.

Jenkins, test this please.

Jenkins CI test (for commit 21aea36f33cf48cd4ad37ec5d5ecf211542b4490, target branch master) succeeded!

Jenkins, test this please.

LGTM!

I'd like to leave this PR not backported to v5 because they contain some divergence on introducing ChainerX.

Could you backport master...21aea36?

Sure!

I've made backport master...7e2d137 since 21aea36 is also a merging commit.


Change EllipsisIcon Glyph

Is there a way to change the EllipsisIcon Glyph while using the NuGet package (i.e. without rebuilding the library)?

From where? The CommandBar?

Sorry, missed your question. I want the EllipseIcon that drops down the secondary commands to be a different icon instead of three dots. The secondary commands will be Shutdown and Restart. Since my app is a headed app for Windows 10 IoT that doesn't include a desktop; the user needs a way to shut down or reboot. The secondary commands on the title bar is the perfect place for it, but I don't see any way to set its glyph. I need the glyph to be the typical power icon instead of three dots.

http://stackoverflow.com/questions/34342022/set-icon-on-secondarycommand-of-commandbar
This has nothing to do with this framework.  The usercontrol PageHeader is a commandbar with extra clothing to make it fit right.   The above link may help get you in the right direction, I think it is what you want but it requires making control Template changes which would be for those that want it for their app, not the framework.    CommandBar is a UWP control therefore the ellipsis is Microsoft default, it is customizable with deep control template monkey business, don't believe its scope is based in this project.  e.g. extreme use case.


Remove default root container in memory DataAccessor + minor fixes

Closes #608
I was wrong in the issue, this probably never worked as I forgot to remove the default container from the memory store when doing the other root-related changes.
Also required changes in some tests which depended on that container being there.
Accidentally discovered a minor bug in the DataAccessorBasedStore when PUTting a resource if there is no root container (which in practice we would never see because of the initializer).

Now to find out why integration tests are failing here but not locally

👍👍 I see it’s time I come back from holidays, this issue involved you having long conversations with yourself 😅

Tbh, it's how I do my best work. 😉


Multiple inputs with different dtype error

When using multiple inputs with different types, torchsummary generates random inputs with same type torch.FloatTensor
You can delete the assignment of dtype , then pass it as a parameter to get differnt random inputs with various types:
    ## del
    # if device == "cuda" and torch.cuda.is_available():
    #     dtype = torch.cuda.FloatTensor
    # else:
    #     dtype = torch.FloatTensor

    # multiple inputs to the network
    if isinstance(input_size, tuple):
        input_size = [input_size]

    # batch_size of 2 for batchnorm
    # modified
    x = [torch.rand(*in_size).type(dtype) for in_size, dtype in input_size]
    # print(type(x[0]))

Then use to get the summary of your model:
summary(model, [((size1), dtype1), ((size2), dtype2)])

What use case would this be for? the current design is to ensure that the user doesn't accidentally pass in certain dtypes as torch.FloatTensor,  and then other types as  torch.cuda.FloatTensor, as that wouldn't work. However, depending on the network architecture, you can't even pass in  torch.FloatTensor and  torch.ByteTensor for multi-inputs, due to the expected input types (eg for conv layers).

So I show the code of my model as following:
class WNGat(nn.Module):
    def __init__(
            self,
            in_channels: int,
            hidden_channels: int,
            out_channels: int,
            heads: int = 1,
            concat: bool = True,
            negative_slope: float = 0.2,
            dropout: float = 0.,
            use_checkpoint: bool = False,
            **kwargs) -> None:

        super(WNGat, self).__init__()
        self.conv1 = GATConv(in_channels, hidden_channels,
                             heads, dropout=dropout)
        self.conv2 = GATConv(hidden_channels*heads,
                             hidden_channels, heads, dropout=dropout)
        self.conv3 = GATConv(hidden_channels*heads,
                             out_channels, heads, dropout=dropout)
        self.use_checkpoint = use_checkpoint

    def forward(self, inp, edge_index, size=None):
        if self.use_checkpoint:
            inp = torch.autograd.Variable(inp, requires_grad=True)
            h1 = F.elu(checkpoint(self.conv1, inp, edge_index))
            h2 = F.elu(checkpoint(self.conv2, h1, edge_index))
            oup = F.elu(checkpoint(self.conv3, h2, edge_index))
        else:
            h1 = F.elu(self.conv1(inp, edge_index))
            # h2 = F.elu(self.conv2(h1, edge_index))
            oup = F.elu(self.conv3(h1, edge_index))
        return oup

Simply, the input of 'GATConv' is a graph where contains node featuressize: (num_nodes, num_node_features) and edges size:(2, num_nodes). For example, there is a graph contains three nodes(0, 1, 2) and each node owns 128-dim features vector:
edges:
    0->1
    1->2
    2->0
    2->2

so the parameters for our model are:
inp: size: [3, 128]  dtype: torch.FloatTensor
edge_index: size: [2, 3]  dtype: torch.Long Tensor  data: [[0, 1, 2, 2],[1, 2, 0, 2]]

In this case, the code would not work as the dtype of inp and edge_index are changed to FloatTensor.
In order to make the code to fit my model, I change some lines in torchsummary.py locally:
## 1 - line 49
 # if device == "cuda" and torch.cuda.is_available():
    #     dtype = torch.cuda.FloatTensor
    # else:
    #     dtype = torch.FloatTensor

    # multiple inputs to the network
    if isinstance(input_size, tuple):
        input_size = [input_size]

    # batch_size of 2 for batchnorm
    x = [torch.rand(*in_size).type(dtype) for in_size, dtype in input_size]
    # print(type(x[0]))

## 2 - line 100
    if input_size and len(input_size[0]) > 1 and type(input_size[0][0]) is tuple:
        total_prod = 1
        for input_item in input_size:
            total_prod *= np.prod(input_item[0])
        total_input_size = abs(np.prod(total_prod) * batch_size * 4. / (1024 ** 2.))
    else:
        total_input_size = abs(np.prod(input_size) * batch_size * 4. / (1024 ** 2.))

Finally, I call the function summary as following to get the right results:

model = model.to(device)
summary(model, [((data.num_nodes, data.num_node_features),
                 torch.FloatTensor), ((2, data.num_nodes), torch.LongTensor)])

----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
           GATConv-1                  [-1, 512]         734,208
           GATConv-2                  [-1, 256]         131,328
================================================================
Total params: 865,536
Trainable params: 865,536
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 80174.01
Forward/backward pass size (MB): 0.01
Params size (MB): 3.30
Estimated Total Size (MB): 80177.31
----------------------------------------------------------------

I find a pull request which maybe work in my case.

Passing in the either dtypes or device type explicitly seems like a popular request, so it's something I would like to add, as this will also address other issues such as being able to run on a specified gpu, eg #92 . However, based on your use case, that PR actually isn't sufficient, as it only allows you to pass in one dtype, and then forces all inputs to that dtype. In your example, you have both torch.FloatTensor, and torch.LongTensor. I have a branch named issue_102 up, let me know if that addresses your issue! I was unable to run your code sample, as its complete since   GATConv is undefined.

Thanks. It worked.


http3: ContentLength must be -1 for unknown body length

According to the http.Request docs, on a server request, a ContentLength of -1 indicates an unknown length.
So in the case of a request without an explicit Content-Length header, the ContentLength was being set to 0, which explicitly indicates there's no body to read. Whereas -1 would indicate to read until EOF.
From my understanding, requestFromHeaders() is only used within the server context, and if it were used for the client context, this would be wrong, since a 0 is correct as unknown length on a client Request.
Please correct me if I am misunderstanding, but this caused me a bug integrating an http3.Server into my project.
I worked around it on my end by doing something similar in my handler:
if req.Body != nil && req.ContentLength == 0 && req.Header.Get("Content-Length") == "" {
    req.ContentLength = -1
}

This is less than ideal, but works in my case.

According to the http.Request docs, on a server request, a ContentLength of -1 indicates an unknown length.

I think I remember that this only applies to requests that actually have a body. Is that correct? What does the Go standard library?

On a server request, Request.Body will never be nil, according to the docs, so even on a GET request, there is a non-nil body, it just immediately returns EOF when attempting to read.
I can put together a small demo if you'd like, but reading the docs on the Request struct make it pretty clear: https://pkg.go.dev/net/http#Request

// For server requests, the Request Body is always non-nil
// but will return EOF immediately when no body is present.
// The Server will close the request body. The ServeHTTP
// Handler does not need to.

// ContentLength records the length of the associated content.
// The value -1 indicates that the length is unknown.
// Values >= 0 indicate that the given number of bytes may
// be read from Body.

Also based on the integration tests failing, I'm not sure of the more widespread ramification of this within the library, but I think to adhere to spec, this is the correct approach. Other stuff internally might need to account for -1?

On a server request, Request.Body will never be nil, according to the docs, so even on a GET request, there is a non-nil body, it just immediately returns EOF when attempting to read.

That doesn't necessarily mean that the value needs to be -1 though. I could also see how 0 would make sense in that case (no body, no length).

Sure it does, according to the ContentLength spec:

The value -1 indicates that the length is unknown.

If I'm implementing a server, and I want to read a body, I would use req.ContentLength to say, allocate a fixed []byte to read my body into. If it's 0, I can't exactly differentiate between "unknown" and "0" without also double checking against the Content-Length header.
But either way, this behavior differs from the stdlib http server, and caused an issue on my end that I needed to work around. I'm fine keeping the workaround, but it feels incorrect on http3.Server's side, according to their docs.

I guess what I'm saying is this isn't about the 0 length case, it's about an unknown length explicitly. If I'm a client, and I send a request without a Content-Length header, but I'm streaming in a long body, saying req.ContentLength of 0, to me as a server handler, indicates the body is a length of 0, rather than an unknown length, as -1 would indicate.

I agree with that case (unknown length of the body). We should definitely fix that.
I'm just saying that we should figure out how to handle the case where there's no body.

I'm just saying that we should figure out how to handle the case where there's no body.

Yeah, I just think that's entirely tangential.
Poking around at what http stdlib does, I think there's a bit of magic happening, since it seems to fill a correct ContentLength in normal cases, even when there's no Content-Length header, so my assumption is it's reading the body into a buffer and then it knows the actual length.
The only case req.ContentLength is coming through as -1 is when Transfer-Encoding: chunked is set on the request headers.
So to be fair, there could be a bug elsewhere, since I'm also using the http3.RoundTripper, and when using that, somehow a Content-Length isn't being set while using an http2.RoundTripper, it is.
I'm going to continue investigating, but I still believe defaulting here to -1 is more correct than 0.

Oh, and I guess to clarify my current findings, I am sending the request as Transfer-Encoding: chunked, which is why through normal http.Server, I get a ContentLength of -1, whereas through http3.Server, I get 0. So maybe we can also check the Transfer-Encoding header.

In stdlib, I guess the logic here is much more in depth: https://cs.opensource.google/go/go/+/refs/tags/go1.19.3:src/net/http/transfer.go;l=669-737;drc=8c17505da792755ea59711fc8349547a4f24b5c5
I'm still inclined to say, sending back -1 when unknown is better than 0. I think the difference is an efficiency gain for clients when they know the length, they can allocate buffers and whatnot to read it efficiently.

Ok, so I really think this boils down to simply being Transfer-Encoding chunked. It seems to be correct in other cases.

Chunked encoding only applies to HTTP 1.1, there's no such thing in HTTP/2 and H3.
The relevant code snippet in HTTP/2 seems to be this one: https://cs.opensource.google/go/x/net/+/refs/tags/v0.2.0:http2/server.go;l=2135-2154
If I understand correctly, this:

doesn't set the content length if there's no body
parses the content length header, if it exists
sets -1 otherwise

Do you agree with my reading of that code? Is this the logic we should implement here as well?

Ah, that makes more sense. I got the Transfer-Encoding because I was using the httputil.DumpRequest, which apparently isn't fully accurate. :) This has been a bit tricky to debug I'll admit.
But it seems that logic is correct to me. I'm not sure off hand how to know in http3 if there is a body or not. Happy to look into it later, or you can commandeer this PR since I'm sure you'll be faster with it.

Maybe we can go by the request method? There's a nice list here: https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/GET.
It looks like only POST, PATCH, DELETE and PUT (please double check) may have request bodies.

Personally I wouldn't do that since the specs allow bodies on other methods, even if they're not typically supported. That's just a choice your implementation can make, say, not supporting a body on a GET. I haven't checked this against what stdlib stuff does since it's not very applicable to me.
I'd say if anything, it's an easy iteration and should be sufficient for a lot of cases.
I wonder if the inverse would be problematic tho? Since you could POST/PUT whatever without bodies at all. So in theory if I POST with no body, I'd expect ContentLength to be 0, not -1. But I think we're splitting hairs here.

@marten-seemann I have updated this with more tests, and to add a set of methods with expected bodies.
This gets us I think to where we wanted, where a GET request ends up with a ContentLength=0, and a POST without a Content-Length header, gets us to ContentLength=-1.

@marten-seemann :wave: let me know if you have any other suggestions or opinions here.

@marten-seemann reaching out here again. I'd like to remove my hacky middleware to work around this issue if I can. :)

I believe this is now resolved with the numerous changes regarding the Content-Length header in the recent releases. Please lmk if it's not.


[FLINK-31343] Remove JMH dependency in flink-table-store-micro-benchmark

JMH open source license is incompatible

Thanks @JingsongLi  LGTM +1


fixes stack deploy to publish ports

This updates the stack deploy command to publish ports.

I'm not sure I understand, shouldn't the bundle file not be able to affect public ports?

  Currently it's not publishing anything.  I'll take another look.  Thx

This appears to be working in master.  User error -- sorry for the distraction.


Add support iSCSI persistent volumes

Description of changes:
Support high availability block volumes that can be mounted by ISCSI protocol to kubernetes pods. Ceph or Longhorn support ISCSI protocol, this PR can help users to mount HA volumes to their EKS clusters by default.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Signed-off-by: Maksim Paskal<EMAIL_ADDRESS>

@maksim-paskal Can you provide some more details about how a user would leverage this ISCSI package in Ceph (or otherwise)? We're reluctant to add such software to this AMI unless it's applicable to a majority of our users, and this looks a bit niche to me.

@cartermckinnon, thanks revieving my PR.
Sure, this PR will help EKS users to use cloud native storages in their clusters, for example Longhorn (or otherwise), without any additional worker configuration on all nodes in cluster. This storages use ISCSI protocol to mount their volumes to pods, to mount volumes, worker needs additional OS packages was installed before mount.
Current scenario installing EKS worker, contains package nfs-utils that needed to mount NFS volumes to pods, I decide if AMI now contains ability to connect to NFS storages, why users can't connect to ISCSI volumes by defaults.
This PR add support for mounting ISCSI volumes in EKS clusters

Thanks for the details! There's a clear case for NFS, because it's needed by the EFS CSI driver; which many, many EKS users utilize. My understanding is that comparatively few of our users roll their own block storage, and these users also tend to run custom AMIs. We try to strike a balance between making this AMI "general purpose" and "universal", taking into account that every piece of software we ship in the AMI requires ongoing support and maintenance (read: security updates).
I'll let others chime in; I'm not opposed to the change if there's sufficient interest.


CFFL

CFFL

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_ccl9up_knw/clone/recipe/meta.yaml:16: pip_install_args: pip install should be run with --no-deps.
ERROR: /tmp/abs_ccl9up_knw/clone/recipe/meta.yaml:21: missing_wheel: For pypi packages, wheel should be present in the host section
ERROR: /tmp/abs_ccl9up_knw/clone/recipe/meta.yaml:55: missing_dev_url: The recipe is missing a dev_url
ERROR: /tmp/abs_ccl9up_knw/clone/recipe/meta.yaml:55: missing_documentation: The recipe is missing a doc_url or doc_source_url
WARNING: /tmp/abs_ccl9up_knw/clone/recipe/meta.yaml:55: missing_description: The recipe is missing a description
Errors were found

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_2dw5u3w6re/clone/recipe/meta.yaml:16: pip_install_args: pip install should be run with --no-deps.
ERROR: /tmp/abs_2dw5u3w6re/clone/recipe/meta.yaml:21: missing_wheel: For pypi packages, wheel should be present in the host section
ERROR: /tmp/abs_2dw5u3w6re/clone/recipe/meta.yaml:55: missing_dev_url: The recipe is missing a dev_url
WARNING: /tmp/abs_2dw5u3w6re/clone/recipe/meta.yaml:55: missing_description: The recipe is missing a description
ERROR: /tmp/abs_2dw5u3w6re/clone/recipe/meta.yaml:55: missing_documentation: The recipe is missing a doc_url or doc_source_url
Errors were found


Install dependencies on forge build

Running forge build automatically installs git modules (forge install) before compiling.
It would be easier if it attempted to install Soldeer packages as well.
Running forge soldeer install on a codebase that doesn't use Soldeer results in an error, but in the case of forge build it could exit silently.
$ forge soldeer install
🦌 Running soldeer update 🦌

Error:
Failed to run soldeer Could not read the config file /path/to/codebase/foundry.toml

This feature request would be best directed at the foundry project! Thanks


ganache-cli not getting installed on Ubuntu

I am getting the following error while trying to install npm install ganache-cli
npm ERR! code ENETUNREACH
npm ERR! errno ENETUNREACH
npm ERR! request to https://registry.npmjs.org/ganache-cli failed, reason: connect ENETUNREACH 2606:4700::6810:1323:443

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/rahul/.npm/_logs/2020-08-15T08_50_50_947Z-debug.log

I executed the following commands in order.
sudo snap install node --classic
sudo npm install -g ganache-cli

Environment
OS : Xubuntu 20.04
Solidity version used : 0.4.24
Node js version 12.18.3
npm version : 6.14.6

Okay, I have resolved it by doing the following.
npm config set registry http://registry.npmjs.org/

Just FYI: you should (almost) NEVER install anything from npm with sudo.


Implement a linter to enforce code style

We need to bring in a tool like eslint to make sure we maintain a certain style for this codebase. I'd like the initial implementation to be very basic and discussions surrounding rules can take place in this issue.
Acceptance Criteria:

Running yarn lint should lint the codebase and print errors where they are found.

I created a PR for basic linting, forgot to mention here I was working on it 😅

cool deal @ncaron I'll take a look at it now!


chore(javascript): add browser playground with instantsearch

🧭 What and Why
🎟 JIRA Ticket: https://algolia.atlassian.net/browse/DI-2904
contributes to https://github.com/algolia/algoliasearch-client-javascript/issues/1551
Changes included:
setup a browser playground with is to casually test things out on the browser/compatibility

nice, maybe it should be in another folder, like instantsearch test

I have a tech debt ticket to add browser tests, I guess I can include instantsearch in it too


[iOS] App Check instructions

Hello.
After installing Firebase Auth on iOS side I see that I can't actually login into the app.
I receive error messages from App Check saying "App is not registered".
I did not setup any checks on my firebase for "App Check" so it is unexpected error.
Please updated README and provide instructions on how to proper setup Firebase Auth on iOS side.

@Monabr can't reproduce this issue. I think it is related something to Firebase side, and not related to the library


Problem: no easy way to setup project dependencies for nix users

Solution: add a shell.nix which'll pull down all the needed
dependencies such as openssl, rust, cargo pkgconfig etc

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@sjmackenzie I realized that this file (which is the same as https://github.com/ethereumproject/sputnikvm/blob/master/shell.nix) does not work if the pkgs parameters is provided. The simplest way is to remove the top-level fn. I'll get a PR for SputnikVM.

See https://github.com/ethereumproject/sputnikvm/pull/203

Please add some nodes to README how to use it as an alternative way.

Will do!


Allow building extensions without recompiling av

Any attempt to from av.packet cimport Packet and build shows a failure that looks like this:
Error compiling Cython file:
------------------------------------------------------------
...
cimport libav as lib
       ^
------------------------------------------------------------

/home/jorge/.local/lib/python3.8/site-packages/av/packet.pxd:1:8: 'libav.pxd' not found

Error compiling Cython file:
------------------------------------------------------------
...
cimport libav as lib

Overview
It looks like while the base pxd files are included, there are some missing which will cause a build failure. If libav.pxd and other files in include were included in the distribution somehow this would allow someone with the correct ffmpeg headers to build extensions on top of pyav without having to fully build pyav itself.
Expected behavior
Build should succeed
Actual behavior
Build fails
Build report:
I'm not actually building pyav from source, i'm using a pip installed version and trying to use the pxd files included in the pip package.
Investigation
I looked in site-packages/av and found that libav.pxd is completely missing.
Reproduction
Run this in a jupyter notebook:
%%cython
from av.packet cimport Packet

Versions

PyAV 12.0.0

Research
I have done the following:

[x ] Checked the PyAV documentation
[ x] Searched on Google
[x ] Searched on Stack Overflow
[x ] Looked through old GitHub issues
[ ] Asked on PyAV Gitter
[ ] ... and waited 72 hours for a response.

@WyattBlue This seems at minimum like a bug. The current pyav distribution incldues pxd files which are impossible to use, so they should either be removed or fixed.

@WyattBlue This seems at minimum like a bug. The current pyav distribution includes pxd files which are impossible to use, so they should either be removed or fixed.
*edit: the current pypi distribution that is

I don't have the admin permissions to modify or remove any release on the av namespace. I'm labeling this as a feature request instead of a bug because, to my knowledge, no av release ever had this behaviour before and I don't think anyone expected this before.
The next step for this would be to find a minimal Cython project that demonstrations the settings needed to do this. Then a PR could be made.


Use case: legal documents

Data set: https://github.com/TextDB/textdb/wiki/Data-Sets
Team members: Derek Fisher, Chang Liu, Jun Ma

@DerekStoneFisher @chang27 @jma19 : please given an update of this issue with the latest status.

have got samples..

@jma19 Please do NOT close the issue until it's done.

After creating a few other related issues, this issue can be closed.


Re-enable IP address usage hiding

Description
With this commit, listUsageRecords filters out usages made on IP addresses on networks with "Hide IP Address Usage". This was previously introduced in #3235.
This requires a DB schema change, to introduce a flag is_hidden in table cloud_usage.cloud_usage.
Cf. #4193
Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[x] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

How Has This Been Tested?
Multiple IP addresses have been created.

Those on shared networks without "Hide IP Address Usage" generate usages normally;
Those on shared networks with "Hide IP Address Usage" generate usages with flag is_hidden, and are not returned by the API.

@blueorangutan package

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2004

@blueorangutan test

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Kindly ping @rhtyd

@blueorangutan package

@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2087

@olivierlemasle Do you have a smoke or integration test for this? it would help guard the feature in the future.
@blueorangutan test

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@DaanHoogland I must say I do not know CloudStack smoke tests well; are there any smoke tests on usages data in CloudStack currently, that I could use as a basis?

look in the test/integration/ subdirectory. I imagine there are marvin tests for related functionality, if you want to go that way. there is https://github.com/apache/cloudstack/blob/master/test/integration/smoke/test_usage.py and I hav e to admit I don't know how relevant it is for your case.

@olivierlemasle which milestone version should it merged? we can set it to 4.16?

@svenvogel as it is marked a bug I wouldimagine 4.15 or before to be appropriate. unless we decide it is too risky and not major..?

:+1: for 4.15! Without it, IP addresses will generate usages on networks flagged "Hide IP address usages"...

@blueorangutan package

@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2255

@blueorangutan test

@DaanHoogland @svenvogel I just rebased on master to fix the conflict in schema-41400to41500.sql.

@blueorangutan package

@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2315

@blueorangutan test

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@andrijapanicsb what do you think about this?

what is being different with this vs the original PR?

@andrijapanicsb
In #3235, usages were filtered out on the fly by the ApiResponseHelper, each time a listUsageRecords result was returned. It means that that filter ran only for the requested page of usages.
However, like all listXxx requests, listUsageRecords's response has a count field which is equal to the total number of usages (all pages included). It then caused the regession #4191, which was fixed in #4193 by reverting the filtering.
To filter the "hidden" IP address usages with a correct count, it is necessary to know how many usages were filtered out in the other pages. This PR moves the filter from the API layer to the usage server, to get the filter right without damaging performances.
NB: Since today's rebase, a timeout in smoke tests cause Travis build to fail.

Trillian test result (tid-3110)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 32514 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4327-t3110-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Smoke tests completed. 86 look OK, 0 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

@DaanHoogland I've just updated the PR after your comment.

@blueorangutan package

@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Verified fix:
Created a VM in a shared n/w with hiding IP Usage enabled, and one in a shared network without hiding IP usage -  3 records were created for both networks, While in all there are 33 records, listusagerecords filters out listing ip usage for the n/w with hide ip usage enabled.
select count(*) from cloud_usage\G
*************************** 1. row ***************************
count(*): 33
1 row in set (0.00 sec)

list usagerecords startdate="2020-10-26 00:00:00" enddate="2020-11-02 00:00:00"
{
  "count": 30,
  "usagerecord": [
    {....
    }
  ]
}

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2340

@blueorangutan test

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-3133)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 33882 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4327-t3133-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_diagnostics.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Smoke tests completed. 86 look OK, 0 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File


Memory changes without a clock edge

See the example here.  It's of an SDRAM controller.  In the example, though, some of the bank state information (kept in a memory structure) changes apart from a clock pulse changing it.
Dan

Closing because example download link is now broken, i.e. can't reproduce.


Tests fail on Windows in gfocce

The automated GitHub tests fail on Windows hosts for all versions of Python. The problem appears to be in the executed of the test for gfocce.
unittests/test_f_j.py::test_gfevnt PASSED                                [ 45%]
unittests/test_f_j.py::test_gffove PASSED                                [ 45%]
unittests/test_f_j.py::test_gfilum PASSED                                [ 45%]
unittests/test_f_j.py::test_gfocce
Error: Process completed with exit code 1.

The two tests that are failing on windows are test_gfocce and test_gfoclt.  I will look to see if I can figure out what's going on with these using only github.

When running both these tests (separately) with a flag saying to actually show what's in standard output, I get
SPICE(BADSUBSCRIPT): Subscript out of range on file line 1626, procedure "zzgfocu". Attempt to access element 1 of variable "svbrad".

A traceback follows. The name of the highest level module is first.
gfocce_c->GFOCCE->ZZGFSOLV->ZZGFOCST

Essentially the same message from gfoclt.
Do we know if there has been any source change in this last several months?  I'm looking at zzgfocu.c and not seeing anything on line 1626 that looks remotely reasonable.

The NAIF source code never changes unless they release an entire new version.

There was no change to our source code between 10/29 (the tests passed)
and 11/5 (the tests failed).  I'm investigation if it's possible to install
an older version of the C compiler (or to find an older github runner) to
see if that's really the issue.
On Wed, Dec 6, 2023 at 3:54 PM Robert French @.***>
wrote:

The NAIF source code never changes unless they release an entire new
version.
—
Reply to this email directly, view it on GitHub
https://github.com/SETI/rms-cspyce/issues/151#issuecomment-1843879118,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABC2LIJQEC6X2LBMVX6YR7LYIEATLAVCNFSM6AAAAABAGNCGEGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBTHA3TSMJRHA
.
You are receiving this because you were assigned.Message ID:
@.***>

Just to gather some notes:
Run #310.  October 29.  Windows-latest 3.11.  Last successful run
Log
Version: 20231024.1.0
Included Software
C compiler: C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.35.32215\bin\HostX86\x64\cl.exe
Run #311.  November 4.  Windows-latest 3.11.  Fail
Log
Version: 20231029.1.0
Included Software
C compiler: C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.37.32822\bin\HostX86\x64\cl.exe
The version and included software are from expanding "Set up Job" > "Runner Image" in the log.  The C compiler was determined by running:
python3 -c 'import distutils.ccompiler as cc; x = cc.new_compiler(); x.initialize(); print(x.cc)'

This document shows the difference between the windows images of 10/24 and 10/29.  There does not appear to be any changes that would affect us.
I do not understand why the name of the C compiler (as reported by Python) is so different between the two Windows images.  Those version numbers do not appear anywhere in the lists of included software.
Looking at the Python requirements between the two logs, the only difference is that one loads in charset_normalizer 3.3.1 and the other loads in 3.3.2.  The problem still occurs even if I force 3.3.1 into the image.
I'm really not sure how to continue at this point


Having no joy with this component - "exports is not defined"

I have a new sveltekit project, I have added svelte-pdfjs (using: npm -i svelte-pdfjs), but my page with the component which is taken more or less from your  routes/index.svelte just gives me an error 500:-
exports is not defined
ReferenceError: exports is not defined
    at http://localhost:3000/node_modules/svelte-pdfjs/node_modules/pdfjs-dist/lib/pdf.js?v=96c44fcc:24:23

I can see pdfjs-dist in my node_modules @ 2.12.313. I've tried earlier version but still get the same.
My page.svelte contains:
<script lang="ts">
	import { Document, Page, PDFJS, preferThisHeight } from 'svelte-pdfjs';
	import type { MultipleOf90 } from 'svelte-pdfjs';
	import workerSrc from 'pdfjs-dist/build/pdf.worker.js?url';
	import { browser } from '$app/env';
	import { selectedPart } from '$lib/stores';

	import { MEDIA_URL } from '$lib/variables';

	if (PDFJS.GlobalWorkerOptions) {
		PDFJS.GlobalWorkerOptions.workerSrc = workerSrc;
	}

	let scale: number = 1;
	let num: number = 1;
	let maxPages = 1;
	let renderTextLayer: boolean;
	let targetHeight = 500;
	let rotation: MultipleOf90 = 0;

	const file = `${MEDIA_URL}${$selectedPart.partfile}`;
	console.log(file);
</script>

{#if browser}
	<Document
		{file}
		on:loadsuccess={(e) => console.log((maxPages = e.detail.numPages))}
		on:loaderror={console.log}
	>
		<div>
			<Page
				{scale}
				{num}
				{renderTextLayer}
				{rotation}
				getViewport={preferThisHeight(targetHeight)}
			/>
		</div>
	</Document>
{/if}

What am I missing? Thanks.
Hamish

@squeezeboxhuf can you check which version of svelte-pdfjs you're using? I was experimenting with some stuff on the alpha versions so if you have those it'll not work. The latest stable version 0.3.4 should work.

I have made some progress! I discovered that I had to have a sveltekit project with TypeScript, then I modded my svelte.config.js and tsconfig.json to match your additional aliases, then copied the component parts from this repository src/lib into my src/lib folder and Hey Presto, I see a PDF document!

I'm glad you found a way to make it work somehow but that's not how the package is meant to be used.
If it's not too much of a bother to you, can you install version 0.3.4 with npm i<EMAIL_ADDRESS>pdfjs-dist and see if the issue persists? Thanks. 🙂

Yes that did the trick! So I reverted my tsconfig and svelte.config, removed the src/lib additions and I am still getting PFs displayed! Thank you.
One extra question! I have some multi-page documents, so I am using a {#each} loop for maxPages numbers to repeat the  component within the  ... like this...
```

{#if browser}
<Document
{file}
on:loadsuccess={(e) => console.log((maxPages = e.detail.numPages))}
on:loaderror={console.log}
>
{#each Array.from({ length: maxPages }, (_, i) => i + 1) as num}
page {num} || rotate

<Page
{scale}
{num}
{renderTextLayer}
{rotation}
getViewport={preferThisWidth(targetWidth)}
/>

{/each}

{/if}

But this results in an error after the first page (first page displays OK) : "PageCanvas.svelte? [sm]:9 Uncaught (in promise) TypeError: Cannot set properties of undefined (setting 'width')  at render_page (PageCanvas.svelte? [sm]:9:17)". So I put the each loop around the <Document> component and it's OK. Effectively I'm creating a <Document> with a <Page> within it, for each page. Is that what you'd expect to be needed? My first presumption was to iterate just the <Pages> element.

I've discovered it might be my page separator paragraph interfering... It's OK to iterate just the <Page> as long as I don't put any other objects in the loop... I've tried putting separator para before and after but it still seems to upset the subsequent pages.

For now, I've left it iterating each containing<Document> and <Page>.

Thanks

Yes - that's great. Thank you.


AutoRest: fix cmd to use autorest.java with preview version

Please check with this argument. I think "autorest-java" and the new "use" is mutually exclusive and you need to verify that.
    const generatorPath = args['autorest-java']
        ? `--use=${path.resolve(args['autorest-java'])} `
        : '';


FileNotFoundError:No such file or directory: './corpus/TCM/dict.txt'

No description provided.

Hi，how did you solve this problem? Thx.


Feature/configurable version support

The new tuyaapi npm package has support for newer devices by setting the protocol version to 3.3, this is something I needed for the newer tuya devices I got.
This small PR adds the following features:

Updates color-convert, debug, mqtt, tuyapi to the latest versions
Allows setting protocolVersion through config.json
Is backwards compatible (defaults to protocol version 3.1, when none has been set)

Thanks for the PR but I already have a build, available here,  that fully supports protocol 3.3 and updates to newer packages as well.  Your PR would only work for cases where all devices in an environment were a single protocol which I think is less than ideal as a lot of people will have devices with different protocols (I'm one of them!).
The code I have written allows specification of the protocol on a per-device level and also supports auto-IP discovery which doesn't require specifying the protocol at all (the discovery packets include the protocol version) so will potentially make configuration easier (you don't even have to specific the IP address).
I've had a number of users testing my build for a while with no issues (once we got the memory leak in tuyapi 5.1.1 fixed) but I want to clean up and simplify the topic configuration a little more before I merge it.  I'll try to get that done next week, but thanks for submitting the PR and supporting the project.

Ah yes, I figured you could run multiple instances for different protocol versions. I'll take a look at your PR when its ready. Taking a quick peek at your readme, I do have a question. Is it also possible to not use discove but still work with Protocol version 3.3?

Yes, it is.  Since you no longer need to "socket" or "lightbulb" versions of the topic (they are still accepted for legacy compatibility) I just overloaded that field so that you can specify the version like so:
tuya/ver3.3////state
tuya/ver3.3////command
If you don't specific the version, but do specify the IP address, the current default is protocol 3.1, but you can always add ver3.1 or ver3.3 to the topic to specify a version, even when using discovery.
I'm not sure this is the best place, and it may change slightly before release, but it's proven to work at this point.  Happy to hear suggestions on that, but I think it's good for temporary.
My future plan is to require a devices.json file which specifies the device info and configures them on each startup.  This will mean we no longer have to be dependent on retain for the MQTT topics and makes it easier to do things like define a friendly name, etc., which will allow for auto-discovery.  That way the topics will be consistent and it won't matter what the protocol is as long as it's defined in the devices.json file.


readme: Added more usage information.

The README in its current state is very helpful. However, I found
myself looking through the application code to get a better
understanding of how the application works, and its relationship
with S3 buckets.
This commit adds information that I think others may find useful
when setting up the application.

Looks good, thanks a lot!


⚠ Cioos Pacific CKAN has degraded performance

In f62c8e1, Cioos Pacific CKAN (https://catalogue.cioospacific.ca) experienced degraded performance:

HTTP code: 200
Response time: 14472 ms

Resolved: Cioos Pacific CKAN performance has improved in 5df9e0c after 56 minutes.


Double check handling of trusted setup by bindings

Two things to check here:

Are bindings freeing the trusted setup correctly? Are they calling free_trusted_setup() and also freeing the pointer (see https://github.com/ethereum/c-kzg-4844/issues/172).
Are we correctly protecting the bindings from double-freeing the trusted setup?

For example, the nodejs bindings do something like this:
https://github.com/ethereum/c-kzg-4844/blob/cc10ef024d7ccf5876eb97e2ad03705373eb6ef5/bindings/node.js/src/kzg.cxx#L43-L48
Should we also be freeing data->settings? Should we also set data->settings to NULL after doing so?

Similar question for nim bindings:
https://github.com/ethereum/c-kzg-4844/blob/cc10ef024d7ccf5876eb97e2ad03705373eb6ef5/bindings/nim/kzg.nim#L34-L35
Should it be freeing x.val?  And setting it to NULL? Or is nim somehow doing it?
(/cc @jangko )

Similar question for nim bindings:
Should it be freeing x.val?

yes, it's tested in test_kzg_ex.nim (not in test_kzg.nim)

And setting it to NULL? Or is nim somehow doing it?

no need, x.val is not a pointer, but a pass by reference type. we can call it 'nim somehow doing it'.

Should we also be freeing data->settings? Should we also set data->settings to NULL after doing so like the Java bindings are doing?

The KzgAddonData struct stores the KZGSettings as a member.  I think when we free the struct it will cleanup that member too.  At least that is the way I understand it, but I also know you know c way better than I do so i say that with pause.
As for setting it to NULL that does seem like a good practice.  On the next line L#47 we free the whole struct though so that NULL member value will be invalid almost immediately.   I would honestly leave that decision up to you and @jtraglia though, and will be happy to add the line setting the member value to NULL if either of you feel it is best for it to be there.

As a note, the addon data is stored as a pointer on the Env object that is managed by the runtime.  Node is structured so that a "context handle" gets passed to all function calls.  The Env class is the n-api representation. Its the magic behind a few things but in this context its for JS allocations.  The result from a JS perspective is developers get garbage collection and lexical scoping.
The cleanup function gets run by the runtime when the Env object is garbage collected at environment tear-down.
So in theory the KzgAddonData pointer will be valid until the runtime is ready for it to be free'd on shut down, and for the same reason @xrchz mentioned here, it is passed back to the implementer to be cleaned up.  Technically there "should not" be a time when the KzgAddonData struct or its members will be referenced once the cleanup function is called.
Empirically this is the behavior that I witnessed.  There is output that happens when the node bits and pieces shut down and the logs from inside the cleanup function were almost to the end, but not necessarily the last line, at program completion.

So in theory the KzgAddonData pointer will be valid until the runtime is ready for it to be free'd on shut down, and for the same reason @xrchz mentioned here, it is passed back to the implementer to be cleaned up. Technically there "should not" be a time when the KzgAddonData struct or its members will be referenced once the cleanup function is called.

typedef struct {
  bool is_setup;
  KZGSettings settings;
} KzgAddonData;

Ah I see. It also seems like settings is not a pointer; it's an actual KZGSettings struct, so there is no need to manually free settings.

FWIW, I only had questions about the node bindings and the nim bindings which have been answered.
@jtraglia should we close this ticket? Do you want to do any investigation yourself? If not, feel free to close this.

Yeah let's close this.


[Proposal] New migrate primitives to resolve failed migrations

tl;dr
Migrations can sometimes fail and this usually leads to situations tricky to solve. Some solutions offer automatic rollbacks but we believe they give a wrong sense of security as rollbacks might or might not succeed.
As we still wanted to improve our support around failed migrations, we decided to propose some new tooling around that and this document explains what we have in mind.
Please read through the proposed solution below and help us understand if this would be helpful to you.
For feedback, simply leave a comment. Alternatively, book 30 minutes with us.
Example of a failed migration
Imagine the following scenario: You created a migration with prisma migrate dev which was saved in a *.sql file containing several SQL statements. These statements are:
ALTER TABLE `Users` DROP `birthDate`;
ALTER TABLE `Users` ADD CONSTRAINT  `unique_emails` UNIQUE (`email`);
ALTER TABLE `Users` ADD COLUMN `signup_date` Date;

Applying the migration succeeded in your dev environment and now it is time to release to production. Unfortunately, the operation fails, since there is non-unique data in your production database (e.g. two users with the same email address).
Migrate returned the error directly from the database which lets you know about the violation of the unique constraint.
ERROR 1062 (23000): Duplicate entry<EMAIL_ADDRESS>for key 'unique_emails'

You now need to recover manually from the partially executed migration ("partially" as the removal of birthDate from table Users was successfully executed).
Automatic rollbacks
Other tools sometimes auto-generate rollbacks. The problem with rollbacks is that they would not be of help here as your migration partially applied already. You now might argue that transactions could have helped but unfortunately transactions are not supported across different databases for DDL statements (SQL statements altering the schema). Additionally, they also come with performance penalties especially on larger datasets.
Proposal: Migrate Primitives
We could offer migrate primitives to resolve failed migrations. Fundamentally you have two options how to proceed, forward and backwards:

fix forward: apply the failed part of the migration or
fix backwards: abort the migration and go back to the state before trying to run the failed migration

The two new commands we are using to fix the scenario from above are:
prisma db execute
# Applies a SQL script to the database without interacting
# with the prisma migrations table.
prisma db execute --target myDB --sql script.sql

prisma migrate diff
# Diffs two diffable targets (prisma schema, database schema or
# migrations history). Output transforms one schema into another
# and can be human-readable or an executable SQL script.
prisma migrate diff --from $PRODUCTION_DATABASE_URL --to prisma/schema.prisma --output fix-migration.sql

Move Forward
The existence of non-unique data in your database is unintentional and you want to fix that. Once fixed, you would be able to go ahead with the migration as planned:

You need to figure out which rows contain duplicate values
You need to delete / alter them to be able to apply the unique constraint
You then can continue applying the rest of the failed migration

You migrate diff from: prodDB to: prismaSchema  output: migrationScript
You db execute the resulting script against your prod database
You db migrate resolve —applied to mark the previously failed migration as succeeded in prod

Your local migration history now yields the same result as the state your prod db is in.
Move Backwards
You realize that non-unique data is valid and you cannot move forward with the migration with the unique constraint included:

You need to create a migration that takes your prod db to the state of your datamodel before the last migration
You get the previous datamodel from git, or you get it from the local migrations history

You migrate diff from: prod to: prev datamodel output: migration script
You db execute that migration against prod.
You db migrate resolve —rolled_back the failed migration marking it as rolled back.
You now delete the folder of the failed migration to prevent it from being run locally again.

Your local migration history now yields the same result as the state your prod db is in. You can now modify the datamodel again to create a migration that suits your new understanding of the feature you're working on (with non-unique mail addresses).
Conclusion
Resolving failed migrations is hard as every situation is different. Often, there is no automatic fix available through rollbacks. We believe offering the described commands gives users the right level of control to address their specific needs.
We can't wait to hear what you think.
For feedback, book 30 minutes with us or leave a comment.
Thanks
Edit 1: Updated diff example

@floelhoeffel thank you for working on this. I'll try to mirror my use-cases against this without claiming to match your knowledge about the underlying workings of Prisma.
When thinking of rollbacks two use cases come to my mind that would be really useful for me:
1. Resolving failed or partially applied migrations
The is described solution would meet my needs for resolving those situations well. If I understood the proposal correctly, this is likely to add a scripts folder to the local prisma directory. These scripts don't have an order to apply them in and are only useful when the target database is in certain states. To me, this would mean that these scrips should be short-lived and shouldn't necessarily be checked into version control (or live inside the project at all).
2. Working on different features
When working on different branches, I often have to apply a certain migration to develop a feature (e. g. add a column). When working on another branch, I need a way to quickly revert the database to the state of the new branch. Using the proposed solution, I would:

Diff the schema on the current branch against the state of the db
Apply the resulting migration script to the database
This sounds great to me, but I am, again, left with an artifact (the migration script) that now lives in my project and is purely situational (it should probably be deleted after application).

A simple solution to this could be to have a simple flag that removes the script after it is applied (this is probably just a detail at this point).
Two additional thoughts that came into my mind are:

AFAIK the concept of a target database does not yet exist in Prisma. Currently, most Prisma applications connect to a database using a connecting string supplied by an environment variable. Does this proposal also include a way to dynamically select a target database based on environment variables?
Will diffing be able to resolve implicit and explicit m2m relations (implicit join tables)?

Again, thank you for your work on this. I'm really excited.

This proposal looks great to address the issue I opened around SQL diffs https://github.com/prisma/prisma/issues/7738
I just have one question about the prisma migrate diff command. Do we imagine it possible to diff between two different formats, such as prisma migrate diff —from mysql://... —to schema.prisma —output pretty|script.sql?
Or would it be restricted to using the same diffable types for the --from and --to arguments? I think allowing the use of different types would make the proposal more flexible using the diff command outside of the failed migrations scenario described.

hey @florianmartens - please find my answers inline

Resolving failed or partially applied migrations
The described solution would meet my needs for resolving those situations well. If I understood the proposal correctly, this is likely to add a scripts folder to the local prisma directory. This is the only caveat I see since these scripts don't have an order to apply them in and are only useful when the target database is in certain states. To me, this would mean that these scrips should be short-lived and shouldn't necessarily be checked into version control (or live inside the project at all).

We envision the output of diff to be short-lived. So you either pipe it into another command or you persist it locally. However we believe there is no good use case for keeping these scripts around. If you want a schema to be reflected in version control the answer is a migration and not the output of diff.

Working on different features
When working on different branches, I often have to apply a certain migration to develop a feature (e. g. add a column). When working on another branch, I need a way to quickly revert the database to the state of the new branch. Using the proposed solution.

Same as above: You would execute the output of diff against your database and move on. There should not be any impact on the migrations in your current branch.

AFAIK the concept of a target database does not yet exist in Prisma. Currently, most Prisma applications connect to a database using a connection string supplied by an environment variable. Does this proposal also include a way to dynamically select a target database based on environment variables?

Great question - we could allow passing in a connection string directly into diff instead of just using whats in schema.prisma. We wouldnt try solve environments with this proposal. That is a separate feature.

hey @maoosi - we envision diffable to support:

Prisma schema files
Migrations history (e.g. files in your migrations folder)

maybe allowing to specify up to which migration e.g. HEAD-1 or something similar

any database we support

Diffing between databases would work as long as the connector is the same. E.g. you could diff a MariaDB and a MySQL database as they use the same prisma connector under the hood.

Hi, I think this proposal is a very good idea. I'm not super deep into how prisma works under the hood.
But coming from #8056 just a quick question: Why the need for a running database to generate the actual migration schemas?
In our current setup the database lives in a private network with the REST endpoints we deployed and developers don't have direct access. Currently we all spin up a local DB when we made changes to the prisma schema and then generate the actual migrations. With a possibility to run e.g. prisma migrate dev --create-only --force-schemas to only create the schemas which then can be checked in to version control this process could be automated in a precommit hook or something similar.
I really appreciate all the work you put into this awesome tool, thanks a lot!

@m1212e - thanks for the input and two quick questions:

what would be the effect of --force-schemas?
how do you apply the migrations to your prod database if the devs dont have access to it? Would be awesome if you could describe that workflow end2end 🙇

That was just a suggestion, you guys will probably find a better name :D But basically it will generate the actual migration files, without requiring a running database to connect to. Currently I need to start an empty mysql db locally only to generate the migration files. With the flag enabled, it would just assume that the db type specified should be used to create the sql files and output them into the migration directory.
Sure: So with our current setup we got our endpoints and database inside a network on our servers. The firewall blocks all access besides http(s). Whenever a  dev makes changes to the schema it runs through all tests etc. and if it isn't breaking anything and no test fails, the ci/cd server deploys and then sends an http request to a specific endpoint, which is starting the migration of the production database. That way we have direct feedback of potentially failing migrations inside our ci/cd.
The reason for this process is beyond my knowledge, actually I found it pretty cumbersome.
My plan for migrating to prisma initially was to upload the whole prisma cli to that http endpoint and then call migrate indirectly via http. A migration API would be helpful but is probably beyond the scope of this issue.

@m1212e

That was just a suggestion, you guys will probably find a better name :D But basically it will generate the actual migration files, without requiring a running database to connect to. Currently I need to start an empty mysql db locally only to generate the migration files. With the flag enabled, it would just assume that the db type specified should be used to create the sql files and output them into the migration directory.

This would be solved by this proposal!

Sure: So with our current setup we got our endpoints and database inside a network on our servers. The firewall blocks all access besides http(s). Whenever a dev makes changes to the schema it runs through all tests etc. and if it isn't breaking anything and no test fails, the ci/cd server deploys (not sure how, probably ssh but devs are denied access) and then sends an http request to a specific endpoint, which is starting the migration of the production database. That way we have direct feedback of potentially failing migrations inside our ci/cd. The reason for this process is beyond my knowledge, actually I found it pretty cumbersome. My plan for migrating to prisma initially was to upload the whole prisma cli to that http endpoint and then call migrate indirectly via http. A migration API would be helpful but is probably beyond the scope of this issue.

Very interesting, thanks for sharing. If you happen to learn more about the reasoning and/or if you wanted to share more about whats cumbersome to you, I am more than happy to jump on a call or read more here. Thanks!

We've spend some time fleshing out the design for the proposed commands. Below is what we plan to implement for the initial version. Please let us know if you have any more feedback.
prisma db execute
Helptext
Preview Feature (Warning)
Run a database native database command on the specified database. You need to specify the datasource and the command input.
The output of the command is connector-specific, and is not meant for returning data, but only to report success or failure.
On SQL databases, this command takes as input a SQL script. The whole script will be sent as a single command to the database.
This command is currently not supported on MongoDB.
Examples:
# Standard input as source and explicit database url.
$ echo 'SELECT `hello world`;' | \
  prisma db execute \
  --preview-feature \
  --stdin \
  --url="mysql://home/testdb"

# SQL file as input and database url from schema.
script.sql
 'SELECT `hello world`;'

$ prisma db execute \
  --preview-feature \
  --file-path ./script.sql \
  --schema schema.prisma

Arguments

The datasource:

--url: the URL of the datasource to run the command on.
--schema: path to the Prisma schema file to take the datasource URL from.
Exactly one of --url or --schema must be provided. None is an error, and more than one is also an error.

The command input:

--file-path : the path towards the file containing the commands
--stdin : take input from stdin
Exactly one of --file-path or --stdin must be provided. None is an error, and more than one is also an error.

Output

Free form, depending on the connector.
Exit code:

0: ok

26 statements executed in 1.4s? (MySQL only)

1: error (red)

Error Handling

engine will error on mongoDB for example

prisma migrate diff
Helptext
Compares the database schema from two arbitrary sources, and displays the differences either as a human-readable summary (by default) or executable script (For SQL databases: SQL, not supported on MongoDB).
[Link to expanded use cases in docs]
prisma migrate diff is a read-only command that does not write anything to any database (except the shadow database in special cases, see below).
A diff command takes a source (--from) and a destination (--to) and compares the source with the destination to generate the diff. The diff can be interpreted as generating a migration that brings the source schema (from) to the shape of the destination schema (to). The following targets are supported:

--from-url / --to-url : a connection string pointing to a live database.
--from-empty /--to-empty : an empty schema (takes no parameter)
--from-datamodel/ --to-datamodel: the datamodel in a schema file (takes a path).
--from-schema/--to-schema: the database specified via the datasource in a schema file (takes a path).
--from-migrations/ --to-migrations: a migrations directory path used with Prisma migrate.

prisma migrate diff will ask you to provide a --shadow-database-url argument when necessary. For example when you diff migration directories with --from-migrations or --to-migrations, Prisma needs a shadow database to determine a schema from the migration files. You can read more about shadow databases at [link]
Examples:
Starting point options:
--from-empty
--from-schema prisma/schema.prisma
--from-datamodel prisma/schema.prisma
--from-url postgres://....
--from-migrations ./migrations...

End point options:
--to-empty
--to-schema prisma/schema.prisma
--to-datamodel prisma/schema.prisma
--to-url postgres://....
--to-migrations ./migrations...

Shadow db:
optional in most cases
--shadow-database-url "$SHADOW_DB"

Output format:
defaults to human readable summary on stdout
--script will render a sql script to stdout

Examples

# From database to database as summary
# e.g. compare two live databases
$ prisma migrate diff \
  --preview-feature \
  --from-url "$DATABASE_URL" \
  --to-url "postgresql://login:password@localhost:5432/db2"

# From empty to current datamodel as a SQL script
# e.g. to squash migrations
$ prisma migrate diff \
  --preview-feature \
  --from-empty \
  --to-datamodel schema.prisma \
  --script

# From current database specified in the schema to empty as a SQL script
# e.g. to teardown a database
$ prisma migrate diff \
  --preview-feature \
  --from-schema schema.prisma \
  --to-empty \
  --script

# From a live db to a Prisma datamodel
# e.g. roll forward after a migration failed in the middle
$ prisma migrate diff \
  --preview-feature \
  --shadow-database-url "$SHADOW_DB" \
  --from-url "$PROD_DB" \
  --to-datamodel=next_datamodel.prisma \
  --script

# From a live database to a datamodel
# e.g. roll backward after a migration failed in the middle
$ prisma migrate diff \
  --preview-feature \
  --shadow-database-url "$SHADOW_DB" \
  --from-url "$PROD_DB" \
  --to-datamodel=previous_datamodel.prisma \
  --script

# From migrations directory to another database
# e.g. generate a migration for a hotfix already applied on production
$ prisma migrate diff \
  --preview-feature \
  --shadow-database-url "$SHADOW_DB" \
  --from-migrations ./migrations \
  --to-url "$PROD_DB" \
  --script

Arguments

Diff targets:

--from / --to

schema — url from schema, takes an optional file path argument, but defaults to the project’s schema.prisma
datamodel — datamodel from schema, takes an optional file path argument, but defaults to the project’s schema.prisma
empty — an empty database
url — the schema of the database behind the url
migrations — can also target up to a specific migration (later)

--shadow-database-url:

required when you use --from-migrations or --to-migrations
or are on a cloud provider for example

--script

defaults to best-effort human-readable summary of the diff
if --script is passed - a script executable on the datasource to bring from --from to --to

Output

The summary or script goes to stdout. This is the only output in the success case.
Exit code:

0: ok
1: error (red)

This proposal is now closed as we just shipped migrate diff and db execute in Preview with 3.9.0.
See the releases notes here
We are looking forward to learning how they work for you and how we could make them better.
You are welcome to post feedback here
And feel free to open an issue if you find a bug 🐛


Globstrings on folders don't work (anymore)

I updated to version 1.5 and I'm getting the following error:
ERROR! Path does not exist: ~/Downloads/*.pdf
Config is:

folders: '~/Downloads/*.pdf'

Seems like globstrings don't work anymore in folders.

Yep this is a bug. Should be fixed in the next version. I added an integration test to prevent regressions in the future. Thank you for reporting!

Thank you for fixing!

New version is just released 👍


Module: terraform-aws-tf-state

Module Repository
voidsolutionsorg/terraform-aws-tf-state
DCO

[X] I sign this project's DCO

Fixed!