Is fast rope exactly equivalent to llama's apply_rotary_pos_emb?

Is fast rope exactly equivalent to llama's apply_rotary_pos_emb? I constructed a test case and found that the result is not exactly equivalent. Is there anything wrong with my case
code:

BS = 2
seq_length = 4
head_num_q = 2
head_num_k = head_num_q
head_dims = 4

def rotate_half(x):
    """Rotates half the hidden dims of the input."""
    x1 = x[..., : x.shape[-1] // 2]
    x2 = x[..., x.shape[-1] // 2 :]
    return torch.cat((-x2, x1), dim=-1)

def apply_rotary_pos_emb(q, k, cos, sin, position_ids, unsqueeze_dim=1):
    cos = cos[position_ids].unsqueeze(unsqueeze_dim)
    sin = sin[position_ids].unsqueeze(unsqueeze_dim)
    q_embed = (q * cos) + (rotate_half(q) * sin)
    k_embed = (k * cos) + (rotate_half(k) * sin)
    return q_embed, k_embed

query_states = torch.rand(BS, head_num_q, seq_length, head_dims, device=0)
key_states = torch.rand(BS, head_num_k, seq_length, head_dims, device=0)
cos = torch.rand(seq_length, head_dims, device=0)
sin = torch.rand(seq_length, head_dims, device=0)
position_ids = torch.arange(0, seq_length, device=0)
position_ids = position_ids.unsqueeze(0).view(-1, seq_length)

q_emb, k_emb = apply_rotary_pos_emb(query_states, key_states, cos, sin, position_ids)
from unsloth.kernels.rope_embedding import fast_rope_embedding
q_emb_fast, k_emb_fast = fast_rope_embedding(query_states, key_states, cos, sin)
print(q_emb)
print(q_emb_fast)

output:
tensor([[[[-0.4640, -0.2159,  1.2940,  0.5061],
          [ 0.2510,  0.6659,  0.7157,  0.7676],
          [-0.3357, -0.1534,  1.0423,  0.2901],
          [-0.0551,  0.4544,  0.3271,  0.4292]],

         [[ 0.1047, -0.0284,  0.7272,  0.6612],
          [ 0.1477,  0.5076,  0.7133,  0.6428],
          [ 0.2858,  0.3088,  0.1674,  0.5372],
          [ 0.5481,  0.7310,  0.1625,  0.6781]]],

        [[[-0.1462,  0.1645,  0.4502,  0.0084],
          [ 0.1149,  0.6139,  0.4423,  0.7918],
          [-0.6824,  0.1163,  0.8660,  0.2474],
          [-0.0140,  0.4709,  0.1222,  0.4574]],

         [[-0.6453,  0.1273,  0.5891,  0.0220],
          [-0.3913,  0.2752,  0.2875,  0.6957],
          [ 0.2637,  0.5442,  0.7180,  0.6723],
          [ 0.6071,  0.3514,  0.2053,  0.3471]]]], device='cuda:0')

tensor([[[[-0.4640, -0.2159,  0.8979,  0.2751],
          [ 0.2510,  0.6659,  0.8333,  0.5259],
          [-0.3357, -0.1534,  1.1098,  0.5500],
          [-0.0551,  0.4544,  0.5717,  0.2453]],

         [[ 0.1047, -0.0284,  0.6065,  0.5705],
          [ 0.1477,  0.5076,  0.8688,  0.4856],
          [ 0.2858,  0.3088,  0.3744,  0.3917],
          [ 0.5481,  0.7310,  0.3588,  0.3084]]],

        [[[-0.1462,  0.1645,  0.3166,  0.1445],
          [ 0.1149,  0.6139,  0.5300,  0.6084],
          [-0.6824,  0.1163,  0.6892,  0.2078],
          [-0.0140,  0.4709,  0.2145,  0.3419]],

         [[-0.6453,  0.1273,  0.2877,  0.1257],
          [-0.3913,  0.2752,  0.5189,  0.7731],
          [ 0.2637,  0.5442,  1.0501,  0.3234],
          [ 0.6071,  0.3514,  0.4423,  0.2948]]]], device='cuda:0')

I  change
Q = Q.view(batchseq_len, n_headshead_dim) --> Q = Q.reshape(batchseq_len, n_headshead_dim)
beacuse of error:
File "/opt/conda/lib/python3.10/site-packages/unsloth/kernels/rope_embedding.py", line 79, in forward
Q = Q.view(batchseq_len, n_headshead_dim)
RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.
Is that the reason?

Hi @danielhanchen,
I test the time consumption of the fast_rope_embedding and the original rope_embedding.
It seems the fast_rope_embedding takes  10 times time than the original rope_embedding.
original RoPE embedding time: 50.787327 new RoPE embedding time: 561.564697
here is the code:
`BS = 2
seq_length = 4096
head_num_q = 32
head_num_k = head_num_q
head_dims = 128
def rotate_half(x):
"""Rotates half the hidden dims of the input."""
x1 = x[..., : x.shape[-1] // 2]
x2 = x[..., x.shape[-1] // 2 :]
return torch.cat((-x2, x1), dim=-1)
def apply_rotary_pos_emb(q, k, cos, sin, position_ids, unsqueeze_dim=1):
cos = cos[position_ids].unsqueeze(unsqueeze_dim)
sin = sin[position_ids].unsqueeze(unsqueeze_dim)
q_embed = (q * cos) + (rotate_half(q) * sin)
k_embed = (k * cos) + (rotate_half(k) * sin)
return q_embed, k_embed
device = torch.device("cuda:2")
query_states = torch.rand(BS, head_num_q, seq_length, head_dims, device=device)
key_states = torch.rand(BS, head_num_k, seq_length, head_dims, device=device)
cos = torch.rand(seq_length, head_dims, device=device)
sin = torch.rand(seq_length, head_dims, device=device)
position_ids = torch.arange(0, seq_length, device=device)
position_ids = position_ids.unsqueeze(0).view(-1, seq_length)
with torch.cuda.device(device):
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)
start.record()
q_emb, k_emb = apply_rotary_pos_emb(query_states, key_states, cos, sin, position_ids)
end.record()
torch.cuda.synchronize(device)
print(f"original RoPE embedding time: {start.elapsed_time(end):.6f}")
with torch.cuda.device(device):
start_fast = torch.cuda.Event(enable_timing=True)
end_fast = torch.cuda.Event(enable_timing=True)
start_fast.record()
q_emb_fast, k_emb_fast = fast_rope_embedding(query_states, key_states, cos, sin)
end_fast.record()
torch.cuda.synchronize(device)
print(f"new RoPE embedding time: {start_fast.elapsed_time(end_fast):.6f} ms")`


Support audio / video

It's not clear to me how / whether Markdown or HTML could support a direct link to audio or video. python-pptx, however, definitely can.
(Personally, I don't have a pressing need for this. But if it's easy to use I just might.)

HTML 5 has <audio> and <video> elements. These are perfectly reasonable things for a user to code - even in Markdown (as it can contain HTML).
One slightly tricky thing is the layout dimensions. I think for <video> the width and height attributes could be used for the proportions, rather than being treated as actual pixel values.

This is coming along but it's causing mass refactoring of the image rendering code - including the creation of a parseMedia function. This refactoring has already simplified the code and fixed a few edge cases.

Pushed code that supports . More testing needed. No documentation yet. No  yet.

Pushed further code that supports retrieving video from the web.
<audio> support remains to be done, as does documentation.

In HTML the <audio> element doesn't have height or width attributes. I think it should probably have a square aspect ratio. When I support it I'll code it that way.

<audio> support is done. As is documentation. All on the trunk.


Added moniker insertion feature & unit test

new files : moniker-controller.ts & moniker-controller.test.ts

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: JohnTheFarmer sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

#362

@JohnTheFarmer Thank you for the contribution! We are reviewing your Pull Request right now and will provide feedback based on your changes.

@JohnTheFarmer The changes look good and seem to follow the same patterns that are being used in the other controllers. And thank you for including unit tests. There are some very small changes in the feedback.

@bharney0  I think its a bug #issue145

@JohnTheFarmer we expect these changes to be released sometime this week. You can test your changes on docs-markdown version 0.2.61. The release will likely be this Thursday 04/02.


log df.count of input/metric/output dataframes

For each input, metric, output would be great to log the count of rows processed. ie if input csv had 100 rows then spark log to show input 100 rows, then if metric had some where filter log metric 47 rows..eetc

We report counts for processed outputs with instrumentation and we can add logs of course.
For all other use cases I wouldn't want to burden the process with counts that I don't need as these are very expensive.


AFNetworking 3.0

We're looking at integrating Yammer but we support AFNetworking 3.0 and it was giving us a conflict in the cocoapod.
This includes AFNetworking 3.0 support in the podspec.

Thanks for the PR, I'd like to merge it but I think this will break support for people who are still using AFNetworking 2.0 in their project, since this will also cause a conflict.

I could redo it without updating the AFNetworking APIs (leaving the deprecated ones) and use <= for the AFNetwork version.


CancellationToken for Async versions

Would there be value in allowing an optional cancellation token to be passed for the SerializeToJsonAndWriteAsync() versions and then passing that to jsonTextWriter.FlushAsync()?

Didn't realize FlushAsync accepts such a token - if it does, yes, that would definitley be a good idea.

I'll make a fork and submit a PR

Merged, tnx!


how to convert to miladi date?

hi mr behzadian
very good library 2 use!
but i didnt find any functions to return miladi date, guid me if there is any way!
TnX ;)

PersianCalendar has some methods to convert persian date to gregorian date.

can u pls tell me what the methods are? i couldn't find any!
tnX

GregorianCalendar gregorianCalendar = new GregorianCalendar();
gregorianCalendar.setTime(persianCalendar.getTime());
You can find more here (In persian): http://smartlab.ir/تاریخ-و-تقویم-هجری-شمسی-در-جاوا/


Test cases for Done class and other

Update the Done class implementation
Add test cases for commons, addcommand, addcommandparser, editcommand, editcommandparser
Add test cases for Done, DeadlineDate, and DeadlineTime

Codecov Report

Merging #143 (1c95421) into v1.3 (6bba6dc) will increase coverage by 0.13%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               v1.3     #143      +/-   ##
============================================
+ Coverage     68.93%   69.06%   +0.13%
- Complexity      442      467      +25
============================================
  Files            78       86       +8
  Lines          1471     1555      +84
  Branches        154      168      +14
============================================
+ Hits           1014     1074      +60
- Misses          396      425      +29
+ Partials         61       56       -5

Impacted Files
Coverage Δ
Complexity Δ

src/main/java/seedu/address/MainApp.java
0.00% <ø> (ø)
0.00 <0.00> (ø)

...java/seedu/address/logic/commands/DoneCommand.java
100.00% <ø> (+100.00%)
7.00 <0.00> (+7.00)

src/main/java/seedu/address/model/Model.java
100.00% <ø> (+33.33%)
1.00 <0.00> (-2.00)
:arrow_up:

.../java/seedu/address/model/person/DeadlineTime.java
83.33% <ø> (+16.66%)
8.00 <0.00> (+2.00)

...ava/seedu/address/model/person/UniqueTaskList.java
84.00% <ø> (+3.14%)
21.00 <0.00> (+1.00)

...n/java/seedu/address/commons/core/index/Index.java
100.00% <100.00%> (ø)
10.00 <0.00> (ø)

...java/seedu/address/logic/commands/EditCommand.java
94.31% <100.00%> (+2.65%)
12.00 <0.00> (ø)

...a/seedu/address/logic/parser/AddCommandParser.java
94.73% <100.00%> (-5.27%)
4.00 <0.00> (-1.00)

...rc/main/java/seedu/address/model/ModelManager.java
92.45% <100.00%> (-1.89%)
24.00 <1.00> (+1.00)
:arrow_down:

...c/main/java/seedu/address/model/person/Status.java
91.66% <100.00%> (+37.82%)
9.00 <0.00> (+3.00)

... and 27 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6bba6dc...1c95421. Read the comment docs.


Add placenta and pancreas as organs

Hi @AlanSimmons, please add placenta and pancreas as organs for the SenNet Data Sharing Portal!
Thanks!
(CC @JAH386)

Dependency on refactoring of UBKG
Although the addition of new organs to the ontology source data is straightforward, it will likely not be available until at least the middle of the first week of August 2023.
I am currently refactoring the UBKG. I will need to regression test some of the endpoints of the UBKG API.

Added to source (SimpleKnowledge spreadsheet).

Added to UBKG CSVs. Should be in Dev on Monday August 8.

Dependency on completion of https://github.com/x-atlas-consortia/hs-ontology-api/issues/14

Results of unit testing
Local Development API querying local development neo4j.

@AlanSimmons Thanks for your help here -- is there a pathway for getting this on PROD?

We completed testing in both local dev (unit) and DEV (system). We plan to promote to PROD soon.
@yuanzhou will be moving the neo4j to PROD, so I will let him provide ETA.

@AlanSimmons Thank you very much for the clarification!


[UI] Navbar color change!

The navbar is not looking good in this color. I'd make it transparent with a blurry effect with will definitely be looking good. Give me a try. If you don't like it don't merge 🫠
Screenshots

hey @akshitagupta15june please assign me this task under iwoc'24.

hello, @akshitagupta15june i saw the website and i want to add some nice looking matching theme color. please assign it to me under IWOC'24 :)


adding github action ci/cd

@krook merged as that is the only way to see the action appear in GitHub UI.


Added TravisBuddy

Describe your change:

[ ] Add an algorithm?
[ ] Fix a bug or typo in an existing algorithm?
[ ] Documentation change?

Checklist:

[ ] I have read CONTRIBUTING.md.
[ ] This pull request is all my own work -- I have not plagiarized.
[ ] I know that pull requests will not be merged if they fail the automated tests.
[ ] This PR only changes one algorithm file.  To ease review, please open separate PRs for separate algorithms.
[ ] All new Python files are placed inside an existing directory.
[ ] All filenames are in all lowercase characters with no spaces or dashes.
[ ] All functions and variable names follow Python naming conventions.
[ ] All function parameters and return values are annotated with Python type hints.
[ ] All functions have doctests that pass the automated testing.
[ ] All new algorithms have a URL in its comments that points to Wikipedia or other similar explanation.
[ ] If this pull request resolves one or more open issues then the commit message contains Fixes: #{$ISSUE_NO}.

Hey @mateuszz0000,
Your changes look good to me!
View build log
TravisBuddy Request Identifier: 1e2dc8a0-a956-11ea-9161-4f77e253115e

Hey @mateuszz0000,
Something went wrong with the build.
TravisCI finished with status errored, which means the build failed because of something unrelated to the tests, such as a problem with a dependency or the build process itself.
View build log
TravisBuddy Request Identifier: b3ee01b0-a957-11ea-9161-4f77e253115e

Hey @mateuszz0000,
Your changes look good to me!
View build log
TravisBuddy Request Identifier: c5f4dff0-a957-11ea-9161-4f77e253115e


Fix deprecation warning

Summary
Fixed a deprecation warning because of a nullable parameter to the reflection class constructor by adding and emtyp string as a default.
ReflectionClass::__construct(): Passing null to parameter #1 ($objectOrClass) of type object|string is deprecated
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)

Hi @barryvdh, confirming this fixed the deprecation warning. Thanks.
ReflectionClass::__construct(): Passing null to parameter #1 ($objectOrClass)
of type object|string is deprecated in
/vendor/barryvdh/laravel-ide-helper/src/Console/MetaCommand.php on line 98

Agreed, if the line above
$concrete = $this->laravel->make($abstract);

produces null in $concrete, something else is weird

Superseded by https://github.com/barryvdh/laravel-ide-helper/pull/1351 which gives more details what's going on and does not silently coerce null into ''


Incomplete instruction in this document...

"To require all PowerShell scripts that you download from the internet are signed by a trusted publisher, run the following command in an elevated Windows PowerShell window (a Windows PowerShell window you open by selecting Run as administrator):
Set-ExecutionPolicy RemoteSigned"
We are then presented with"
"[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "N"):"
Which one are we supposed to set, please?
Cheers,
David.

Document details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 39fbe5a2-c138-b566-c094-82ffc3d1b734
Version Independent ID: 4d69ee5d-5a6d-a203-69fe-bf5441a8b5eb
Content: Connect to Exchange Online PowerShell
Content Source: exchange/docs-conceptual/exchange-online/connect-to-exchange-online-powershell/connect-to-exchange-online-powershell.md
Service: exchange-online
GitHub Login: @chrisda
Microsoft Alias: chrisda

@officedocsbot assign @yogkumgit

Hi @deejerydoo, thank you for your feedback.
"[Y] Yes" or  "[A] Yes to All". Both works.

@dariomws  Thank you very much for the contribution and sharing this explanation. @deejerydoo
Hope this comment is helpful for you. If you see a documentation update is required, please feel free to open an issue for the same. We proceed here to close it. Thanks for taking out some time to open the issue. Appreciate and encourage you to do the same in future also.


refactor(v2): improve semantic doc sidebar markup

Motivation
Just make the markup for the doc sidebar a little more semantic.
Refs:

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/aside
https://fae.disability.illinois.edu/rulesets/ROLE_7/
https://html.com/tags/aside/
http://mariechatfield.com/blog/semantic-html

For example, Bootstrap, VuePress, Next.js sites use similar markup.
Have you read the Contributing Guidelines on pull requests?
Yes
Test Plan
Preview (no visual changes, actually)
Related PRs
(If this PR adds or changes functionality, please take some time to update the docs at https://github.com/facebook/docusaurus, and link to your PR here.)

👍


LCFS - Save draft of new transfer

Description:
Develop the functionality to save a draft of a new transfer:

Save Draft button
Requires the three Transfer Details inputs (quantity, select organization, price per unit) to not be blank / be filled out / selected
Requires the user to enter an agreement date in the Agreement date feature, and that date to be today's date or earlier (cannot be a future date)
Quantity cannot be more than the organization has in its current balance minus the units that are in reserve
Clicking the Save Draft button creates the transaction and sets the status to Draft
Creating the transaction assigns an ID number to the transfer (i.e., a transaction ID number, similar to TFRS functionality where every transaction is given a unique ID number, sequential order).
A success alert appears (Material UI examples)
The user is returned to the transaction page where the transaction is visible at the top of the transactions table
The transaction row for the transfer is highlighted in yellow

Validation messages:

Quantity, price per unit, and agreement date fields: "Please fill out this field."
Select an Organization: "Please select a transfer partner."
Not enough compliance units (quantity is more than the organization has in its current balance minus the units that are in reserve): "Your organization does not have enough compliance units for this transfer."

Wireframes & screenshots:

New transfer page (wireframe): https://preview.uxpin.com/59145260c7d8cc48969ac79401cc03d84843c3d9#/pages/162954828/simulate/no-panels
Example of front-end validation (TFRS screenshot): Private Zenhub Image
Example of Transactions table row highlight feature (TFRS screenshot): Private Zenhub Image

Purpose and benefit to user:
Allows the BCeID user to save a draft of the transfer to be processed later and/or by a different user
Acceptance Criteria:

[x] Given I am a BCEID user, should be able to save the transfer to database.
[ ] Given I am a BCeID user, the data I entered should be validated before saving to the database.

Development Checklist:

[ ] Validation framework discussion would be required.

Notes:

Hey team! Please add your planning poker estimate with Zenhub @AlexZorkin @jig-patel @kevin-hashimoto @prv-proton

When attempting to save a draft transfer, a BCeID user with signing authority gets the following error after pressing 'Save Draft' button: "Error occurred while submitting. Please retry. For ongoing issues, contact support".
Please see screen shots below:

Save draft feature is working, but there is no validation that restricts organization from selling more compliance units than they have available.
Still requires the component that notes an organization does not have enough compliance units (quantity is more than the organization has in its current balance minus the units that are in reserve): "Your organization does not have enough compliance units for this transfer."
Please see screenshot that allowed me to selling substantially more compliance units that the organization has:

Validation of organization balance will be addressed on another card #477


Names mixed up in plot_arcs_compare

When comparing two arc plots RNavigate will exchange the names of the two samples.

This also effected the sequence bar. Both have been fixed. Thanks Simon!


Unable to connect to remote server

Trying to execute:
Get-RsFolderContent  –RsFolder '/Sample Reports' -Credential  Connect-RsReportServer -ReportServerUri 'http:///ReportServer'   -Verbose| Format-Table -AutoSize -Verbose
gives the following error:
VERBOSE: Establishing proxy connection to http:///ReportService2010.asmx...
Exception calling "ListChildren" with "2" argument(s): "Unable to connect to the remote server"
At C:\Program
Files\WindowsPowerShell\Modules\ReportingServicesTools\<IP_ADDRESS>\Functions\CatalogItems\Get-RsFolderContent.ps1:85 char:17

            $Proxy.ListChildren($Item, $Recurse)

            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

CategoryInfo          : NotSpecified: (:) [], ParentContainsErrorRecordException
FullyQualifiedErrorId : WebException

Works on a local server; only failing when making calls to remote SSRS server, please advise.

Resolved - the config file on the remote server had to be changed to .
It was set at 0 due to which remote connection was not being made.

Would you be able to tell me exactly what to change? Thank you very much.


Figure release announcement

Bump version and sha256 version

@sbesson  I re-used this PR to update the version of the website
I will merge it when travis is green


Export/import single configuration

Possibility to not only export/import all configurations, but also for just one configuration

Would be easy to share configs this way 😀

It's already available under configure options.


Omega: How should we handle attestations of work completed?

For Omega, we're targeting the top 10,000 projects, using tooling (Omega Analysis Toolchain, etc.) and triage.
We need to provide some evidence that work was completed, both for internal tracking ("did we already look at X?") and external assurance ("has X been reviewed and believed to be safe to use?").
Let's use this thread to discuss how we can do this.
Some preliminaries:
We'll want to assert that some activity took place. That activity could be a tool execution (with some result), a manual action ("reviewing a thing"), or some combination of the two. For either, the target could be a physical artifact (foo.tar.gz) or something else, like a GitHub repo (like Scorecard results).
In-toto attestation seems to be a reasonable vehicle for providing attestations, but delegates the actual predicate content to the user (us) -- though I haven't read the specs in a while, so I might be wrong here.
Some questions:

How granular should the attestations be?

Tool X did not find SQL injection in subject Y?
Tool X did not find any critical issues in subject Y?
Tool X did not find any critical issues in artifact Z associated with subject Y?

What is the appropriate subject? In-toto has opinions on this being a physical artifact (with a hash). What does that mean in the context of: source code in git repo, source archive on Debian, patches applied, compiled binaries for N different architectures? Are (transitive) dependencies included in the mix at all?
How much information should we provide? Give too little, and end-users need to trust the Omega team, perhaps more than they want to. Give too much, and we risk 0-daying the community.

Option A: Only provide attestations of "no issues found" or "issues were found and fixed in version X"
Option B: Only provide aggregated results, clearly specified: (4 potential issues found [untriaged]). Problem there is that it's a trivial lift to go and download and re-run the toolchain to find the details of those 4 potential issues. On the other hand, the tools are public anyway, anyone could run them at any time.

What's the relationship of PackageURL with this?

I came up with a slightly frankenstein'ed attestation format, to see what this could look like. I'll push the code to generate this up shortly.
{
  "_type": "https://github.com/ossf/alpha-omega/omega-analysis-toolchain/Statement/v0.1",
  "_comment": "Generated by the Omega Analysis Toolchain",
  "subject": [
    {
      "type": "https://github.com/ossf/alpha-omega/omega-analysis-toolchain/Types/PackageURL/v0.1",
      "purl"<EMAIL_ADDRESS>      "digest": {
        "alg": "sha256",
        "value": "hwwP4QliI6WNT4gy0Ip+ZR6i/K245od7L9wmtmLUgd0="
      },
      "filename"<EMAIL_ADDRESS>    }
  ],
  "predicateType": "https://github.com/ossf/alpha-omega/omega-analysis-toolchain/Predicate/v0.1",
  "predicate": {
    "review_text": "This was a lot of fun reviewing this, but I didn't find any issues."
  },
  "status": "pass",
  "signature": "MEQCIG3RfjZb/LMjpwSDvapI6TJDzLS/5moghpvWLFyHwZTkAiBJNdCrNrr+rDfaJqk3uMnRCCnKkFRegjbS2sjKyAQSYg=="
}

Thoughts?

https://github.com/in-toto/attestation/issues/77 is relevant here. We've been talking about defining human review predicates for code review probably starting with VCS reviews and dependency reviews like with crev. It'd be great to have alpha-omega predicates as part of this!

Another minor pass at an attestation schema:
{
  "$schema": "https://openssf.org/verde/schema/1.0",
  "timestamp": "2022-10-23T10:26:13-07:00",
  "type": {
    "uri": "https://openssf.org/verde/assertion-uri/no-critical-findings-by-tool",
    "name": "No critical findings by tool",
    "qualification": {
      "tool_name": "https://openssf.org/verde/tools/devskim",
      "version": "0.6.2",
      "command_line": null,
      "ruleset": {
        "default": true
      }
    }
  },
  "subject": {
    "purl"<EMAIL_ADDRESS>  },
  "predicate": {
    "findings": {
      "raw": {
        "critical": 0,
        "high": 0,
        "medium": 0,
        "low": 3
      },
      "triage_complete": false,
      "triaged": {
      }
    }
  },
  "signature": "MEQCIEeRkTzulKg/OeizLK5lrwxV9qKZtDTSqG043R8hB7IyAiBX3P1ivxfQNgZkvaYEp9w9xlWgP0Of90Sy0Y8KC62tXg=="
}

We should also take a look at OSCAL, but from a first glance, it seems very complicated and doesn't actually cover this use case, but I could be wrong there.

{
  "_type": "https://in-toto.io/Statement/v0.1",
  "subject": [
    {
      "name"<EMAIL_ADDRESS>      "url"<EMAIL_ADDRESS> // requires https://github.com/in-toto/attestation/pull/95
      "digest": {
        "sha256": "abcdef..."
      }
    }
  ],
  "predicateType": "https://github.com/ossf/alpha-omega/v0.1",
  "predicate": {
    "review_text": "This was a lot of fun reviewing this, but I didn't find any issues.",
    "status": "pass",
    "findings": {
      "raw": {
        "critical": 0,
        "high": 0,
        "medium": 0,
        "low": 3
      },
      "triage_complete": false,
      "triaged": {}
    },
    "tooling": {
      "name": "https://openssf.org/verde/tools/devskim",
      "version": "0.6.2",
      "rulseset": {"default": true},
      "command_line": null
    }
  }
}

I tried taking a pass at this spec as an in-toto attestation. Note that I've not included the signature itself here because this would be the payload to DSSE, which will include the actual signature. I'm a little unclear on a couple of the fields in your passes but I think some discussion will help clarify that for me!

Thanks @adityasaky! Yep, I like this better, headed in the right direction. In the meantime, I made some other updates, including using the REGO policy engine from OpenPolicyAgent to set up consumption policies on the assertions.
Example assertion (for security advisories):
{
  "_type": "https://in-toto.io/Statement/v0.1",
  "operational": {
    "execution_start": "2022-11-03T19:38:42.195814Z",
    "execution_stop": "2022-11-03T19:38:42.196237Z",
    "environment": {
      "operator": null,
      "hostname": "scovetta-xps",
      "machine_identifier": "00000000-0000-0000-0000-cc96e5042f66"
    }
  },
  "subject": [
    {
      "type": "https://github.com/ossf/alpha-omega/omega-analysis-toolchain/Types/PackageURL/v0.1",
      "purl"<EMAIL_ADDRESS>    }
  ],
  "predicateType": "https://github.com/ossf/alpha-omega/v0.1",
  "predicateGenerator": {
    "name": "openssf.omega.security_advisories[deps.dev]",
    "version": "0.1.0"
  },
  "predicate": {
    "security_advisories": {
      "medium": 4,
      "high": 4,
      "critical": 2,
      "unknown": 10
    }
  },
  "timestamp": "2022-09-08T08:30:01Z",
  "signature": "MEQCIDrirKX+puM8agLVAPxO88kG7zcuI7GK9RMaLnh/bud/AiAcq+AjpYfamogV/JHBiC1ybXw4a9yIhUZNJmxu3YNiRw=="
}

And then the policy that would check for this:
package openssf.omega.policy.no_critical_vulnerabilities

# VERSION 0.1.0
# LAST UPDATED 2022-11-03 Michael Scovetta

default pass = false

# Identify whether this policy applies to a given data object
applies := true {
    input.predicateGenerator.name == "openssf.omega.security_advisories[deps.dev]"
    input.predicateType == "https://github.com/ossf/alpha-omega/v0.1"
}

pass := true {
    input.predicate.security_advisories
    input.predicate.security_advisories.critical == 0
}

I'm certain that I'm not writing the rego policies the "right" way -- still getting my head around the language.
Yep, I agree, whether the signature is DSSE, COSE, or whatever -- it should probably be outside of the main assertion above.

Could you explain the distinction between the execution times in operational and timestamp? Also, I think this information would also go into the predicate itself to conform to the attestation framework.

the policy would check

I'm not very familiar with rego myself but I understand what you're checking for and it makes sense to me.
Here's my pass at the attestation statement which would then go into DSSE as a payload.
{
  "_type": "https://in-toto.io/Statement/v0.1",
  "subject": [
    {
      "type": "https://github.com/ossf/alpha-omega/omega-analysis-toolchain/Types/PackageURL/v0.1",
      "purl"<EMAIL_ADDRESS>    }
  ],
  "predicateType": "https://github.com/ossf/alpha-omega/v0.1",
  "predicate": {
    "generator": {
      "name": "openssf.omega.security_advisories[deps.dev]",
      "version": "0.1.0"
    },
    "security_advisories": {
      "medium": 4,
      "high": 4,
      "critical": 2,
      "unknown": 10
    },
    "operational": {
      "execution_start": "2022-11-03T19:38:42.195814Z",
      "execution_stop": "2022-11-03T19:38:42.196237Z",
      "environment": {
        "operator": null,
        "hostname": "scovetta-xps",
        "machine_identifier": "00000000-0000-0000-0000-cc96e5042f66"
      }
    },
    "timestamp": "2022-09-08T08:30:01Z"
  }
}

Yep, that makes sense! I'll re-work to put it all within the predicate.

Could you explain the distinction between the execution times in operational and timestamp?

I was think of the operational timestamps as being "as the assertion is being created" - I don't know that we need start/end, but some sort of a tie back to the system that created it, as opposed to the systems that generate the outer levels of the assertion.
The "timestamp" was more "at what time were the facts true?" -- example, suppose security advisories are refreshed weekly -- we pull them on 11/7 but they're true as of 11/4. We'd have 11/7 in the operational section and 11/4 in the timestamp.
Maybe we need better names for the fields, but that was my thinking...

@scovetta, @adityasaky - Nice to meet you.

Reproducing results - Do we need to be able to reproduce the result from the attestation? The attestation is currently missing information to reproduce the results.
Raw output - I think a generic Do we enforce expectations on the tools to emit raw results in a standard format like SARIF? or provide a format to reference raw logs or output and format typefindings: {raw: {url: '', type: 'application/json', schema: 'schemaurl'}}
security advisories - Severity standardiz

I think it would be worth adding a hash value for all of the scan artifacts in the operational data. In an ideal world, these attestations would be perfectly reproducible by others. From a more practical perspective, it lets there be a more verifiable connection between these public attestations and private results.

I think it would be worth adding a hash value for all of the scan artifacts in the operational data. In an ideal world, these attestations would be perfectly reproducible by others. From a more practical perspective, it lets there be a more verifiable connection between these public attestations and private results.

Agree. There needs to be a connection between the public attestation and private results.

Agree that hashes of inputs would be valuable here.
I have a couple other incomplete thoughts here:
What if we thought of "reproducibility" as a property that emerges from having multiple attestation authors? If three different trusted parties all provided a "no publicly known vulnerabilities" attestation, does that meet the underlying goal?
Alternatively, how do we feel about reproducibility being less boolean and more nuanced? E.g. for anything where the analysis results could change over time.
I'll keep noodling on this.

Here's another iteration:
{
  "_type": "https://in-toto.io/Statement/v0.1",
  "subject": [
    {
      "type": "https://github.com/ossf/alpha-omega/omega-analysis-toolchain/Types/PackageURL/v0.1",
      "purl"<EMAIL_ADDRESS>    }
  ],
  "predicateType": "https://github.com/ossf/alpha-omega/v0.1",
  "predicate": {
    "generator": {
      "name": "openssf.omega.security_advisories[deps.dev]",
      "version": "0.1.0"
    },
    "operational": {
      "execution_start": "2022-11-07T23:32:21.749384Z",
      "execution_stop": "2022-11-07T23:32:21.749801Z",
      "environment": {
        "operator": null,
        "hostname": "scovetta-xps",
        "machine_identifier": "00000000-0000-0000-0000-cc96e5042f66"
      },
      "timestamp": "2021-12-14T19:30:14Z"
    },
    "content": {
      "security_advisories": {
        "medium": 2
      }
    },
    "evidence": {
      "_type": "https://github.com/ossf/alpha-omega/types/evidence/v0.1",
      "reproducibility": "temporal",
      "source-type": "url",
      "source": "https://deps.dev/_/s/npm/p/express/v/4.4.3",
      "content": [
        {
          "source": "GHSA",
          "sourceID": "GHSA-gpvr-g6gh-9mc2",
          "sourceURL": "https://github.com/advisories/GHSA-gpvr-g6gh-9mc2",
          "title": "No Charset in Content-Type Header in express",
          "description": "Vulnerable versions of express do not specify a charset field in the content-type header while displaying 400 level response messages. The lack of enforcing user's browser to set correct charset, could be leveraged by an attacker to perform a cross-site scripting attack, using non-standard encodings, like UTF-7.\n\n\n## Recommendation\n\nFor express 3.x, update express to version 3.11 or later.\nFor express 4.x, update express to version 4.5 or later. ",
          "referenceURLs": [
            "https://nvd.nist.gov/vuln/detail/CVE-2014-6393",
            "https://github.com/advisories/GHSA-gpvr-g6gh-9mc2",
            "https://www.npmjs.com/advisories/8",
            "https://bugzilla.redhat.com/show_bug.cgi?id=1203190",
            "https://nodesecurity.io/advisories/express-no-charset-in-content-type-header"
          ],
          "severity": "MEDIUM",
          "gitHubSeverity": "MODERATE",
          "scoreV3": 6.1,
          "aliases": [
            "CVE-2014-6393"
          ],
          "disclosedAt":<PHONE_NUMBER>,
          "observedAt":<PHONE_NUMBER>
        },
        {
          "source": "NSWG",
          "sourceID": "NSWG-ECO-8",
          "sourceURL": "https://github.com/nodejs/security-wg/blob/main/vuln/npm/8.json",
          "title": "No Charset in Content-Type Header",
          "description": "Vulnerable versions of express do not specify a charset field in the content-type header while displaying 400 level response messages. The lack of enforcing user's browser to set correct charset, could be leveraged by an attacker to perform a cross-site scripting attack, using non-standard encodings, like UTF-7.",
          "referenceURLs": [],
          "severity": "MEDIUM",
          "gitHubSeverity": "UNKNOWN",
          "scoreV3": 5.4,
          "aliases": [
            "CVE-2014-6393"
          ],
          "disclosedAt":<PHONE_NUMBER>,
          "observedAt":<PHONE_NUMBER>
        }
      ]
    }
  }
}

@scovetta - I think if three well-known independent researchers put out the same attestation for something, that's probably sufficient regarding the attestation credibility. The hash of the artifacts for reproducibility on that front would mainly be just an extra layer of protection in that case. e.g. for situations where two attestations each attest 1 critical finding, but they are attesting different findings.
I think more importantly, though, having a hash of the artifacts gives assurance to anyone who is digging into the attestation, such as publicly attesting to a vulnerability in a piece of open source, then privately going to the authors and giving them the details and artifacts, and the authors can independently verify that these artifacts hash to the same value in the public attestation.

@antrompl - If I understand your perspective, does separating the findings from attestation make sense. Having the attester information + reference to the findings being attested in the payload, will that work? while keeping the findings as a separate in-toto statement or file that can be referred in attestation statement.

About reproducibility, should we not go after the characteristics of the code and the scanner? For example, a tool with a rule set (maybe identified by some hash) was run on a specific repository (with a commit hash) and we found a set of things. Later in order to reproduce, we will have to run the same tool on the same code. If we are running on a different code (different branch, different commit), then we will not be able to reproduce. Maybe these are more important then the time.
The time is more important for recency information. So that an older attestation on a branch will be replaced by a newer attestation (more recent commit hash).

@openrefactory Agree - we should have enough information in the assertion to probably re-create it, but there are a bunch of reasons why this might not be possible:

OS-specific functionality, either on the part of the tool, or the component's build, etc.
Non-deterministic scanning (e.g. fuzzing)
Dependence on external things (web services, etc.)

But we should try our best anyway -- How would:

Subject: Package + URL + Hash, or Repo URL + Branch + Commit?
Tool: Name + Command Line?

What else do you think we should include?

Another round of updates in:

Assertions get stored in a local SQLite database. (Obviously this is temporary.)
Added verification for the assertions via check-assertion-trust.py -- any signatures that don't validate have their assertions ignored. (This is temporary too.)
Reworked the schema a bit:

{
    "_type": "https://in-toto.io/Statement/v0.1",
    "predicate": {
        "content": {
            "file_extensions": [
                "",
                "md",
                "json",
                "js",
                "npmignore"
            ],
            "programming_languages": [
                "package.json",
                "Unknown",
                "javascript"
            ]
        },
        "evidence": {
            "_type": "https://github.com/ossf/alpha-omega/types/evidence/v0.1",
            "content": {
                "output": {
                    "appVersion": "Microsoft Application Inspector 1.6.16-beta+7d93aa8d85",
                    "metaData": {
                        "CPUTargets": [],
                        "OSTargets": [],
                        "appTypes": [
                            "web.application",
                            "web.service"
                        ],
                        "applicationName": "/opt/src/npm/express/4.4.3",
                        "authors": "\"",
                        "cloudTargets": [],
                        "dateScanned": "11/09/2022 22:58:20",
                        "description": "Sinatra inspired web development framework",
                        "detailedMatchList": [
                            {
                                "confidence": "High",
                                "endLocationColumn": 24,
                                "endLocationLine": 6,
                                "excerpt": " */\n\nvar escapeHtml = require('escape-html');\nvar http = require('http');\nvar path = require('path');\nvar mixin = require('utils-merge');\nvar sign = require('cookie-signature').sign;\n",
                                "fileName": "/opt/src/npm/express/4.4.3/reference-binaries/npm-express<EMAIL_ADDRESS>                                "language": "javascript",
                                "pattern": "require\\(['\\\"](http|https|request|axios|superagent|got)",
                                "ruleDescription": "Network Connection: HTTP",
                                "ruleId": "AI032400",
                                "ruleName": "Network Connection: HTTP",
                                "sample": "require('http",
                                "severity": "Moderate",
                                "startLocationColumn": 11,
                                "startLocationLine": 6,
                                "tags": [
                                    "OS.Network.Connection.Http"
                                ],
                                "type": "Regex"
                            },
                            ...
                        ],
                        "filesSkipped": 0,
                        "filesTimeOutSkipped": 0,
                        "filesTimedOut": 0,
                        "languages": {
                            "Unknown": 8,
                            "javascript": 24,
                            "package.json": 2
                        },
                        "lastUpdated": "01/01/0001 00:00:00",
                        "outputs": [],
                        "packageTypes": [],
                        "sourcePath": "/opt/src/npm/express/4.4.3",
                        "sourceVersion": "4.4.3",
                        "tagCounters": [
                            {
                                "count": 44,
                                "tag": "Metric.Code.Function.Defined"
                            },
                            {
                                "count": 2,
                                "tag": "Metric.Code.URL"
                            },
                            {
                                "count": 2,
                                "tag": "Metric.Code.Logging.Call"
                            },
                            {
                                "count": 14,
                                "tag": "Metric.Code.Exception.Caught"
                            }
                        ],
                        "targets": [],
                        "timedOut": false,
                        "totalFiles": 34,
                        "totalMatchesCount": 78,
                        "uniqueDependencies": [],
                        "uniqueMatchesCount": 17,
                        "uniqueTags": [
                            "Application.Type.Web.Service",
                            "CloudServices.Code.Repo.GitHub",
                            "Cryptography.Encoding.Base64",
                            "Cryptography.Encryption.General",
                            "Cryptography.HashAlgorithm.Legacy",
                            "Data.Sensitive.Credentials",
                            "Data.Sensitive.Identification",
                            "Data.Sensitive.Secret",
                            "Metadata.Application.Author",
                            "Metadata.Application.Description",
                            "Metadata.Application.Version",
                            "OS.Network.Connection.General",
                            "OS.Network.Connection.Http",
                            "OS.Network.Connection.Http.Ajax",
                            "OS.Network.Connection.Socket",
                            "OS.Process.DynamicExecution",
                            "WebApp.Cookies.Attr.Expires"
                        ]
                    },
                    "resultCode": 0
                }
            },
            "reproducibility": "high",
            "source": "docker run --rm -t -v /tmp/omega-dee912:/opt/export --env-file .env openssf/omega-toolshed:latest<EMAIL_ADDRESS>            "source-type": "command"
        },
        "generator": {
            "name": "openssf.omega.programming_languages",
            "version": "0.1.0"
        },
        "operational": {
            "environment": {
                "hostname": "scovetta-xps",
                "machine_identifier": "00000000-0000-0000-0000-cc96e5042f66",
                "operator": null
            },
            "execution_start": "2022-11-09T23:01:31.165813Z",
            "execution_stop": "2022-11-09T23:01:31.166015Z",
            "timestamp": "2022-11-09T22:58:20Z"
        }
    },
    "predicateType": "https://github.com/ossf/alpha-omega/v0.1",
    "subject": [
        {
            "purl"<EMAIL_ADDRESS>            "type": "https://github.com/ossf/alpha-omega/omega-analysis-toolchain/Types/PackageURL/v0.1"
        }
    ]
}

Still to do (~today):

[ ] Add the commit or file hash to the subject
[ ] End to end demo / screencast

@antrompl  - My view on attestation is that it is source of security score of a package based on some standard. Attestation identity provides credibility. The findings are the information that the attestation identity used to provide attestation. So keeping it separate makes sense to me.
So the expected behavior from a software consumer is that their policy will be based on the attestation score and not based on the findings. It will be good to define the boundary and separate the information that goes into attestation and security findings.

@scovetta -

nit: missing hash of<EMAIL_ADDRESS>and Application Inspector. Do we need purl for Application Inspector
this goes back to our discussion on should evidence be part of attestation.
What is the attestation part here? Is the attestation that we ran Application Inspector on Express?

Thanks @dasarpjonam!

nit: missing hash of<EMAIL_ADDRESS>and Application Inspector. Do we need purl for Application Inspector

Yes, the subject should contain both the reference (e.g<EMAIL_ADDRESS>as well as the hash of the underlying files scanned.
For the tool, I'm not sure the hash is needed, but certainly a version reference that's enough to get back to it. Though maybe -- I suppose versions aren't always clear/immutable, but I wonder how slippery of a slope this is (do I need to include the architecture of the box the tool was run from? environment variables present? host distro?) -- I'm not sure where to stop -- for PoC/MVP, maybe stopping with "Application Inspector version X" is enough?

Is the attestation that we ran Application Inspector on Express? this is for my own context and clarification.

Yes, "subject" is the target of analysis (express). From the perspective of the assertion above, "Application Inspector" is just an implementation detail -- it could be swapped out in favor of something else, with the assertion remaining the same.

A couple screenshots of current progress:
Running analysis and generating assertions against a package:

One of the resultant assertions:
{
    "_type": "https://in-toto.io/Statement/v0.1",
    "predicate": {
        "content": {
            "scorecard_data": {
                "binary_artifacts": 10,
                "branch_protection": 1,
                "ci_tests": 10,
                "cii_best_practices": 0,
                "code_review": 1,
                "contributors": 10,
                "dangerous_workflow": 10,
                "dependency_update_tool": 0,
                "fuzzing": 0,
                "license": 10,
                "maintained": 10,
                "packaging": -1,
                "pinned_dependencies": 7,
                "sast": 0,
                "security_policy": 10,
                "signed_releases": -1,
                "token_permissions": 0,
                "vulnerabilities": 10
            }
        },
        "evidence": {
            "_type": "https://github.com/ossf/alpha-omega/types/evidence/v0.1",
            "content": {
                "checks": [
                    {
                        "details": null,
                        "documentation": {
                            "short": "Determines if the project has generated executable (binary) artifacts in the source repository.",
                            "url": "https://github.com/ossf/scorecard/blob/6a00f92156aa890436cd1591f8d244cfeb6a5ebf/docs/checks.md#binary-artifacts"
                        },
                        "name": "Binary-Artifacts",
                        "reason": "no binaries found in the repo",
                        "score": 10
                    },
                    ...<snip>...
                ],
                "date": "2022-11-11",
                "metadata": null,
                "repo": {
                    "commit": "8368dc178af16b91b576c4c1d135f701a0007e5d",
                    "name": "github.com/expressjs/express"
                },
                "score": 5.4,
                "scorecard": {
                    "commit": "6a00f92156aa890436cd1591f8d244cfeb6a5ebf",
                    "version": "v4.8.0-45-g6a00f92"
                }
            },
            "reproducibility": "temporal",
            "source": [
                "docker",
                "run",
                "-e",
                "GITHUB_AUTH_TOKEN=***",
                "gcr.io/openssf/scorecard:stable",
                "--format",
                "json",
                "--npm",
                "express"
            ],
            "source_type": "command"
        },
        "generator": {
            "name": "openssf.omega.security_scorecards",
            "version": "0.1.0"
        },
        "operational": {
            "environment": {
                "hostname": "scovetta-xps",
                "machine_identifier": "00000000-0000-0000-0000-cc96e5042f66",
                "operator": null
            },
            "execution_start": "2022-11-11T08:03:43.693406Z",
            "execution_stop": "2022-11-11T08:03:43.693610Z",
            "timestamp": "2022-11-11T08:03:43Z"
        }
    },
    "predicateType": "https://github.com/ossf/alpha-omega/v0.1",
    "signature": "MEYCIQCXDiwBGFvNiIMEpjaBPB+7GMJPLUFNkBw9g+sq/Th+jAIhAJNkRff3a9+ex+6jv50U50iIJ/gNQUbKJQ3/qUsDTpuq",
    "subject": {
        "purl"<EMAIL_ADDRESS>        "type": "https://github.com/ossf/alpha-omega/omega-analysis-toolchain/Types/PackageURL/v0.1"
    }
}

The policy that requires the project be "actively "maintained":
package openssf.omega.policy.actively_maintained__sc

# This policy is used to determine if a project is actively maintained.
# The logic is as follows:
# 1. If the project has any change in the last 6 months, it is actively maintained.
# 2. Else if the project has any code change in the last 12 months, it is actively maintained.
# 3. Else it is not actively maintained.
#
# VERSION 0.1.1
# LAST UPDATED 2022-11-08 Michael Scovetta

default pass = false
default applies = false

# Identify whether this policy applies to a given data object
applies := true {
    input.predicate.generator.name == "openssf.omega.security_scorecards"
    input.predicateType == "https://github.com/ossf/alpha-omega/v0.1"
}

pass := true {
    input.predicate.content.scorecard_data.maintained > 7
}

Since "maintained" has a value of 10, we'll expect this policy to pass when analyzing this assertion:

I have quick comment on the CLI API, on single vs. double dash, see: https://github.com/ossf/alpha-omega/issues/35

Does anyone have thoughts on where we should store assertions for PoC? We have local SQLite and later today a local file system (git repo) will be pushed in.
Do we need an actual running service for get/put operations?

@scovetta can you link me to where you're generating the attestation in the scanner?

Thanks! On a side note, would you be interested in some of these bits (especially base.py) being upstreamed to https://github.com/in-toto/in-toto? In our Go implementation, we define the models for in-toto attestations but not so yet in the Python implementation.

Thanks! On a side note, would you be interested in some of these bits (especially base.py) being upstreamed to https://github.com/in-toto/in-toto? In our Go implementation, we define the models for in-toto attestations but not so yet in the Python implementation.

@adityasaky Of course! I can't guarantee things will be stable on our end yet, but I'd be thrilled to align and share with in-toto.


Add AlexNemmo's file

I want to sleep, but can`t
created a branch, created a file and made a commit, and opened a pull request

Step 7: Respond to a review
Your pull request is looking great!
Let’s add some content to your file. Replace line 5 of your file with a quotation or meme and witty caption. Remember: Markdown is supported.
⌨ Activity: Change your file
1. Click the [Files Changed tab](https://github.com/AlexNemmo/github-slideshow/pull/7/files) in this pull request

2. Edit your newly added file

3. Replace line 5 with something new

4. Scroll to the bottom and click **Commit Changes**

Note : Can't find the button to edit the file? It may look like a pencil, or it may look like three dots.
I'll respond when I detect a commit on this branch.
I want to sleep, but can`t

I want to sleep, but cantI want to sleep, but cant


Changed code signature in version 2.0.2b23?

hello - I've noticed that the code signature has changed between (stable) version 1.10.0b18 and 2.0.2b23:
EXPECTED 'Robots and Pencils Inc. (PBH8V487HB)'
FOUND 'Matt Kiazyk (ZU6GR6B2FY)'

can you confirm this @MattKiazyk? thanks!
To Reproduce
codesign -dv --verbose=4 /Path/To/Application.app

Hi @suschizu
Correct. Because of the change in organizations (see more info here) it is now signed by my App Developer account.
I would love to in the future have an actual XcodesOrg Developer Account, but doing that would take lots of time and money and I'm not sure yet if I want to go that route for a simple utility app.
Let me know if you have any more questions

thanks for the fast answer - we wish you all the best here from corecode :)


[SPARK-30404][Core] Fix wrong log for FetchFailed task's successful speculation

What changes were proposed in this pull request?

When a task fails due to FetchFailedException, its index is recorded in fetchFailedIndex.

Its log is "INFO TaskSetManager: Task ${info.id} in stage ${taskSet.id} (TID $tid) failed (because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run)."

If another task of the same TaskSet finishes successfully (e.g., a speculative task which was launched before FetchFailedException, then succeed), the index will be removed from fetchFailedIndex. And the log of the successful task is:
"INFO TaskSetManager: Finished task 1.1 in stage 1.0 (TID 3) in 10000 ms on <IP_ADDRESS> (executor 2) (2/2)",
instead of
"INFO TaskSetManager: Ignoring task-finished event for 1.1 in stage 1.0 because task 1 has already completed successfully".

If another task of another TaskSet attempt of this stage succeed, the index will also be removed from fetchFailedIndex.

If a task succeed, and then its copy task fails, the log of the latter failed task is
"INFO TaskSetManager: Task ${info.id} in stage ${taskSet.id} (TID $tid) failed (but a different copy of the task has already succeeded)."

Why are the changes needed?
We should output correct and more accurate log information.
Does this PR introduce any user-facing change?
No
How was this patch tested?
test manually, and verify output logs.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

The PR description is improved.

Can one of the admins verify this patch?


fix: add handling for unmanaged files to vacuum command

Description
Adds handling for unmanaged files to the VACUUM command. The VACUUM command should correctly handle files that are not managed by a Delta table, e.g. files and directories that start with _ or ..
Additionally fixes test test_non_managed_files.
Related Issue(s)

closes #227 (Unit and integration tests for the vacuum command)

We may have some additional aspects into consideration when selecting the files to be vacuumed. As it stands I think we may accidentally delete deletion vector files for files that are still part of the log. We should however be able to get the corresponding files names from the add action and add them to the managed files.
Some .crc files might also get deleted pre-maturely, but these are always optional, so no harm to the table data.

I'll pick this up next week since we getting often issues about vacuum not cleaning up all required files.


ARROW-6381: [C++] BufferOutputStream::Write does extra work that slows down small writes

Before:
-------------------------------------------------------------------------------
Benchmark                                        Time           CPU Iterations
-------------------------------------------------------------------------------
BufferOutputStreamSmallWrites/real_time  218998806 ns  218938001 ns         18   584.478MB/s

After
-------------------------------------------------------------------------------
Benchmark                                        Time           CPU Iterations
-------------------------------------------------------------------------------
BufferOutputStreamSmallWrites/real_time   11439597 ns   11436629 ns         61   2.73169GB/s

I found this when looking at Parquet perf data (this is used there, it might be better to use BufferBuilder instead)

Interesting. I wonder if BufferOutputStream should rely on BufferBuilder or if that would degrade performance.

Current benchmarks here:
-------------------------------------------------------------------------------
Benchmark                                        Time           CPU Iterations
-------------------------------------------------------------------------------
BufferOutputStreamTinyWrites/real_time    40346155 ns   40231527 ns         52   793.136MB/s
BufferOutputStreamSmallWrites/real_time   15462451 ns   15438635 ns        100   2.02099GB/s
BufferOutputStreamLargeWrites/real_time   14104049 ns   14066428 ns        178   2.17907GB/s

Codecov Report

Merging #5222 into master will increase coverage by 1.59%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #5222      +/-   ##
==========================================
+ Coverage   87.67%   89.27%   +1.59%
==========================================
  Files        1025      742     -283
  Lines      147030   106158   -40872
  Branches     1437        0    -1437
==========================================
- Hits       128915    94776   -34139
+ Misses      17753    11382    -6371
+ Partials      362        0     -362

Impacted Files
Coverage Δ

cpp/src/arrow/buffer_builder.h
100% <ø> (ø)
:arrow_up:

cpp/src/arrow/io/memory.cc
90.47% <100%> (+0.05%)
:arrow_up:

cpp/src/plasma/thirdparty/ae/ae.c
70.75% <0%> (-0.95%)
:arrow_down:

python/pyarrow/tests/test_parquet.py
96.56% <0%> (-0.07%)
:arrow_down:

js/src/util/fn.ts

go/arrow/memory/memory_avx2_amd64.go

rust/datafusion/src/execution/filter.rs

rust/arrow/src/csv/writer.rs

rust/datafusion/src/bin/main.rs

go/arrow/ipc/file_reader.go

... and 277 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 05bc63c...997158b. Read the comment docs.

For the record, here are the results on a Windows 7 VM. It will be interesting to check if mimalloc improves those numbers:
--------------------------------------------------------------------------------
------------------
Benchmark                                        Time             CPU   Iterations UserCounters...
--------------------------------------------------------------------------------
------------------
BufferOutputStreamTinyWrites/real_time    90775594 ns     90725806 ns 31 bytes_per_second=352.518M/s
BufferOutputStreamSmallWrites/real_time   32634961 ns     32567771 ns 83 bytes_per_second=980.525M/s
BufferOutputStreamLargeWrites/real_time   31290053 ns     31250000 ns 91 bytes_per_second=1005.79M/s

Thanks @pitrou!

Interesting. I wonder if BufferOutputStream should rely on BufferBuilder or if that would degrade performance.

Probably, it seems a bit weird to have two implementations of nearly the same thing. https://issues.apache.org/jira/browse/ARROW-6388

+1


fix logic bug asking for pw while yk is not inserted

when yk is not present hook is supposed to time out after a certain
time. when password challange response mode is enabled it loops asking
for a password until it gets timed out. password for challange response
is only required if yk gets detected.

Good job!


Fix D8 NPE on Gradke Sync in Andoid Studio 4.2 c 14 with kotlin  1.4.…

Fix D8 NPE on Gradke Sync in Andoid Studio 4.2 c 14 with kotlin 1.4.10

@googlebot I signed it!

@googlebot I signed it!


Desktop app not recognizing account type

OS: Windows 10
GPMDP Version:  4.6.0
Issue Descriptions: Today, my account type listed in the desktop app changed from Family Plan to Google Play Music Free, although I did not change my plan and the Google Play Music app on my phone still lists it as Family Plan. I don't know how to get the desktop app to recognize my family plan again. Any suggestions? Thanks!
Steps to Reproduce: Account settings

Sorry, my mistake!


Any thoughts on a Flutter release?

Hi guys
I'm opening this issue just to know if you have any support planned for Flutter;
Flutter has already support for embedding Native iOS/Android views, (You can take Flutter Google Maps as an example).
Flutter projects support iOS Cocoapods and I'm pretty sure they do support additional Android libraries.
The idea would be a port from ViroReact, and I'm pretty sure it's a possible thing to do.
I'd be willing to help :smile:

Hi Kevin, thanks for reaching out and for the offer to help. At this time, we are focused on our two core products, ViroReact and ViroCore (Java). But Flutter is something we will consider for the future if we add more language support. Will definitely hit you up then. Again, thanks for the suggestion and offer to help.

I think it is time for Flutter support :)
Even RN now is taking advantage of Skia


How to make top menu bar items right-justified

Not sure if this makes sense, but the previous version of Hugo Academic included a top menu bar where the site name was left-justified and the links were right-justified. The latest version has them all squished together (see my website).
Is there a way to change back to the older way?
Thanks in advance.

We use GitHub issues for managing the project and have a dedicated forum and Discord community for questions: https://github.com/wowchemy/wowchemy-hugo-modules/discussions
In recent versions, there is an alignment option for main menu links in params.toml:
# Main menu alignment (l = left, c = center, r = right)
main_menu = {align = "c"}


[Windows7] Middle-click in Docs window breaks the UI

Clicking the middle mouse button in the Docs window causes it to go into a "hands-free" scrolling mode but it's impossible to get out of this mode. As a result, left-clicking anywhere else in the UI has no effect. I can right-click on some UI elements and get a context menu which will work but even if I close the Docs window, the rest of the UI is still busted.

I wasn't able to reproduce this just now on a Windows 7 computer running a earlier-today source version.

I can't reproduce this on osx. It's possible this will be resolved once we switch docs to a browser tab #2101. I'll leave this open for a bit longer to see if we get more any more information to reproduce this. If we can't reproduce we can't fix it

Closing as we haven't heard back and I also can't reproduce on ubuntu 14


Service Worker/Home Page App

What is the desired outcome of an effective service worker implementation?
On a Pixel 2 XL (Android 8.1), it works as a home screen app without connectivity (needs a title/icon), but does not within the browser.

I don't have a vision for how this should/could function offline. Maybe @coreyar can chime in.

According to the docs the service works only work in supporting browsers in a production build and over https.
For offline homepage apps we can define a title and icon in public/manifest.json.
If we want to go the extra mile we can display an image to the user that the cache has populated and works offline. This can be done in src/registerServiceWorker.js


Unable to sync with Netflix - won't load trakt.tv watch history

I am using the [Beta (2023-02-02T22:13:42.303Z)] release in Chrome, and I can no longer sync my Netflix watch history. The last time I synced was sometime before trakt.tv had some downtime do to a database crash. I mention that just in case that has something to do with it.
WHen I try to sync my Netflix watch history each show/movie on the "Trakt side" displays "LOADING HISTORY..." forever, it finds the show/movie on trakt but seems unable to find my watch history on trakt. Please see the attached screenshot.

I have the same problem. Plugin does not seem to be able to get the trakt history anymore.


Conflict with Pipenv When Executing Bin Files

Describe the bug
Right away I wish to front load this discussion by saying I don't know if this is indeed a bug with rtx or if it's just my unfamiliarity with the Python ecosystem or a Lovecraftian fusion of both. 😅
With that said, I had recently encountered a bizarre and frustrating situation while attempting to run pytest on a project I had been developing on at work today when this weird bug crept up:
It kept claiming the app's dependencies (e.g., mock, freezegun) were not installed. Thinking it odd but not unheard of, I just shrugged my shoulders and attempted to resync my Pipenv environment. It told me everything was already installed. I then attempted to a hard, clean re-install by deleting the app's local .venv directory before installing everything again.
After some time, it finished installing, so I tried again -- same error!
After some troubleshooting, I realized that all of my installs were being directed towards the rtx (shim?) directory, which is located within my $HOME directory; however, it was running pytest from .venv/bin/pytest, so it had been expecting the dependencies/modules to be located in the project's virtual environment directory. As I'm sure you already guessed, none of them were there.
I had tried a number of things from running rtx shell and pipenv shell --always a number of times, and even tried reshelling my fish.
None of it worked!
Although, I did end up getting it to work, even if I feel to perhaps be nothing more workaround: I adjusted the configuration settings of my project's local .rtx.toml file to have the listed [tools] entry for python to now look like this:
[tools]
python = { version = '3.8.17', virtualenv = '.venv' }

Before, it only had the version number.
And even now, I have to install packages for it using python -m pipenv install <options> after making sure that it's actually using the correct .venv/bin/python path this time.
To Reproduce
Not quite sure how to reproduce it, exactly.
The only things I can think of to mention are having this as your best fitting Python/pipenv project look something like this:
[env]
PIP_NO_CACHE_DIR = 'false'

PIPENV_VENV_IN_PROJECT = 'true'
PIPENV_VERBOSITY = '-1'

PYTEST_THEME = 'github-dark'
PYTEST_MODE = 'dark'

PYTHONDONTWRITEBYTECODE = 'true'
PYTHONFAULTHANDLER = 'true'
PYTHONUNBUFFERED = 'true'

PYTHONPATH = "{{env.PWD}}"
TESTPATH = '{{env.PWD}}/tests/unittests'

[tools]
python = "3.8.17"

I also made it a point to have things like pytest only installed in my respective project's respective virtual environment directories and never globally; however, I did end up having pipenv installed globally, because I thought that'd be safe to do so without it leading to any problems. With that said, even after uninstalling it, I still have to use python -m pipenv just to make sure the dependencies/modules are coming from the right place.
Expected behavior
A clear and concise description of what you expected to happen.
rtx doctor output
rtx version:
  2023.9.1 macos-arm64 (c6d59d7 2023-09-13)

build:
  Target: aarch64-apple-darwin
  Features: DEFAULT, NATIVE_TLS, OPENSSL
  Built: Wed, 13 Sep 2023 14:18:58 +0000
  Rust Version: rustc 1.72.0 (5680fa18f 2023-08-23)
  Profile: release

shell:
  /opt/homebrew/bin/fish
  fish, version 3.6.1

rtx data directory:
  /Users/ooo998/.local/share/rtx

rtx environment variables:
  RTX_SHELL=fish
  RTX_PYTHON_DEFAULT_PACKAGES_FILE=/Users/ooo998/.default-python-packages

settings:
  {"always_keep_download": "false", "always_keep_install": "false", "asdf_compat": "false", "disable_default_shorthands": "false", "disable_tools": "[]", "experimental": "false", "jobs": "4", "legacy_version_file": "true", "legacy_version_file_disable_tools": "[]", "log_level": "INFO", "missing_runtime_behavior": "warn", "plugin_autoupdate_last_check_duration": "10080", "raw": "false", "trusted_config_paths": "[]", "verbose": "false", "yes": "false"}

config files:
  /Users/ooo998/.config/rtx/config.toml
  /Users/ooo998/projects/capitalone/raas/bank-fraud-case-manager/.rtx.toml

plugins:
  go       (core)
  java     (core)
  node     (core)
  python   (core)
  ruby     (core)

toolset:
<EMAIL_ADDRESS>node@latest

No problems found

I figured out the issue here: list_bin_paths adds the bin directory to PATH when activating the virtualenv. This has an unintended side effect which is that it creates a shim for every bin in that directory.
What needs to happen is PATH should be modified without making all of these shims.

Apologies for the slow reply; I've been crazy busy lately.
At any case, thank you so much for looking into it and for the fast fix! :)


fix: cloud account/provider missing event logs for delete/enable/disable/change_project

这个 PR 实现什么功能/修复什么问题:
修正：cloudaccount/provider缺少启用，禁用，删除，更改项目的操作日志
是否需要 backport 到之前的 release 分支:

release/2.8.0

/cc zexi

/cherrypick release/2.8.0

/lgtm
/approve

Codecov Report

Merging #514 into master will decrease coverage by 0.02%.
The diff coverage is 0%.

@@            Coverage Diff            @@
##           master    #514      +/-   ##
=========================================
- Coverage    7.41%   7.39%   -0.03%
=========================================
  Files         345     345
  Lines       54242   54259      +17
=========================================
- Hits         4023    4010      -13
- Misses      49843   49873      +30
  Partials      376     376

Flag
Coverage Δ

#aFlag
7.39% <0%> (-0.03%)
:arrow_down:

Impacted Files
Coverage Δ

pkg/cloudcommon/db/virtualresource.go
0% <0%> (ø)
:arrow_up:

pkg/cloudcommon/db/enabledstatusstandalone.go
0% <0%> (ø)
:arrow_up:

pkg/compute/models/cloudproviders.go
1.23% <0%> (-0.04%)
:arrow_down:

pkg/compute/models/cloudaccounts.go
1.01% <0%> (ø)
:arrow_up:

pkg/appsrv/response.go
66.15% <0%> (-3.08%)
:arrow_down:

pkg/compute/baremetal/diskconfig.go
58.85% <0%> (-1.79%)
:arrow_down:

pkg/compute/models/cloudregions.go
2.29% <0%> (-0.12%)
:arrow_down:

pkg/compute/models/zones.go
2.21% <0%> (-0.08%)
:arrow_down:

pkg/compute/models/hosts.go
0.38% <0%> (-0.01%)
:arrow_down:

pkg/compute/models/cloudproviderregions.go
0.44% <0%> (+0.01%)
:arrow_up:

... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 207b760...ffbb7e3. Read the comment docs.


PoC: Firewalls in AWS ECF Dev are receiving threat policy updates from SecOps

Describe the issue
SecOps has a process to deploy firewall policy updates to their CP firewalls. We want to enable SecOps to deploy threat policy updates to our CP firewalls in AWS ECF Dev
Additional context

SecOps creates hand-built custom threat policies.  Michael V is the author, and he applies these across gov using a tool.  The tool likely won't connect to our firewalls.  We just want to try the threat policies and see how they behave.  The connectivity can happen later.
There are 2 kinds of policies relating to Check Point firewall appliances: network and threat.  The SEA covers us for network at the moment, so we're only interested in the list of threats (we think this is a ban-list of IPs and websites)

Definition of done

Meet with Micheal V from SecOps and provide connection info to our ECF Dev firewall so that he can provide latest threat policy update he wishes whenever he wishes.  This will allow CPF to observe the stability and effect on the cloud environments over time, allowing us to understand how we could set up a pipeline with stability check automation eventually going from Dev -> Test -> Prod ECF Checkpoint Firewalls.
If a direct linkage between Check Point products cannot happen due to network barriers, we may need to do an initial prototype that is something simple like a regular email from him to us, then we manually add it to our AWS Forge environment and observe if anything breaks
SecOps is able to deploy CP firewall policy updates in AWS ECF Dev

We will revisit this once we have direct connect in place

As SECOPS (Chris C.) is taking up firewall support/update/migration, firewall ticket become irrelevant.


Filters component refactor

update tests and code to use shared examples
remove custom accordion and use GDS component instead
move out of shared namespace
add preview component

todo (in another PR)

look into accordion.scss as we shouldnt need this file, any styles we do need should prob live elsewhere
remove dependency of moj-filter
there is complexity can be removed from this component as there are options that we dont use

Review app deployed to https://teaching-vacancies-review-pr-3186.london.cloudapps.digital


feat(@angular-devkit/build-optimizer): strip Angular decorators in __…

…decorate call

/cc @danbucholtz


Use ClearAndInitialize to improve performance

Summarize Functionality
We implemented ClearAndInitialize to improve performance in Get-DbaDbView in #8042.
Background info (https://github.com/sqlcollaborative/dbatools/pull/8042#issuecomment-1003418389):
Write up from SMO team for those interested:
https://github.com/microsoft/sqlmanagementobjects/wiki/Refresh-vs-ClearAndInitialize

ClearAndInitialize offers a parameterized explicit initialization combined with a complete reset of the collection contents. Unlike Refresh, prior object references from the collection are not preserved. For example, if your application had a variable named col that was a reference to a Column object from table.Columns, after table.Columns.Refresh() your reference would be the same as that returned by table.Columns[col.Name]. After ClearAndInitialize the indexer would return a different object. All objects in the collection will have the same set of populated properties after the call, using whatever list had been provided to SetDefaultInitFields plus those provided to the extraFields parameter. It's common for applications using this API to skip calling SetDefaultInitFields and just pass the property names to this function.

Let's use this issue to plan implementing ClearAndInitialize in other Get- commands and for general discussion on this topic.
Is there a command that is similiar or close to what you are looking for?
No
Technical Details
No response

Let me share a small snippet from a code I use when rebuilding all indexes to build page checksums. To be able to have a nice progress bar, I work with DataSpaceUsed and IndexSpaceUsed. To improve performance I get these properties for all tables at once with ClearAndInitialize:
$db = Get-DbaDatabase -SqlInstance $instance -SqlCredential $credential -Database $database
$db.Tables.ClearAndInitialize('', [string[]]('Schema', 'Name', 'HasClusteredIndex', 'DataSpaceUsed', 'IndexSpaceUsed'))

$countTarget = 0
$spaceUsedTarget = 0
foreach ($table in $db.Tables) {
    $countTarget++
    $spaceUsedTarget += $table.DataSpaceUsed + $table.IndexSpaceUsed
}


Can we use S3 Bucket for Rekor attestation storage

Hi,
I have deployed and set up the Sigstore on the private Kubernetes environment.
I have used Mysql and KMS for the database and signing keys.
I would like to use the S3 as a Rekor attestation storage.
I have configured like below. but getting an error as shown below.
bucket: s3://sigstore-rekor
Configuring attestation storage at s3://sigstore-rekor
42
2023-11-20T15:22:33.378Z	PANIC	api/api.go:173	open blob.Bucket: no driver registered for "s3" for URL "s3://sigstore-rekor"; available schemes: file, gs, mem
41
github.com/sigstore/rekor/pkg/api.ConfigureAPI
How can I configure S3? Please let me know..

I don't have an easy way to test this, but #1844 should fix this.

We might want to setup some e2e tests with minio


Screen lock implicitly pauses timer

Hello! When my screen is locked because I've left the machine (meeting, tea, whatever) the timer doesn't decrease, only continuing when the screen is once again unlocked. This throws me off my timings a bit, it would be grand if this could be either turned off or configurable.

What is probably happening is your MacOS automatically sleeps after some time which is causing the timer to suspend. If you simply lock the screen the timer keeps running. I'm not an expert in MacOS programming but it seems to me there might not be a way to keep the timer running while the system sleeps.
There's an option called "Prevent automatic sleeping on power adapter when the display is off" under the Battery system settings, in case you want to try that.

For instance if I start a 20 minute long break at 12:00 and my laptop were asleep from 12:02 to 12:18, I think it's still {reasonable, possible} for the timer to go off at 12:20 rather than 12:36, no?

Got it, I agree, in that case the app should be able to correct the remaining time.


RUSTFLAGS='-C target-feature=+crt-static' cargo build --release --target x86_64-unknown-linux-gnu failed

I want to static build krustlet as follows, but failed:
git clone https://github.com/krustlet/krustlet.git
cd krustlet
git checkout tags/v1.0.0-alpha.1
RUSTFLAGS='-C target-feature=+crt-static' cargo build --release --target x86_64-unknown-linux-gnu
error: generator cannot be sent between threads safely
--> crates/kubelet/src/store/oci/client.rs:61:99
|
61 |       async fn pull(&mut self, image: &Reference, auth: &RegistryAuth) -> anyhow::Result {
|  ______________________________________________________________________________________________^
62 | |         self.pull(image, auth, vec![manifest::WASM_LAYER_MEDIA_TYPE])
63 | |             .await
64 | |     }
| |^ future created by async block is not Send
|
= help: the trait Sync is not implemented for core::fmt::Opaque
= note: required for the cast to the object type dyn futures::Future<Output = Result<ImageData, anyhow::Error>> + std::marker::Send
error: generator cannot be sent between threads safely
--> crates/kubelet/src/store/oci/client.rs:70:33
|
70 |       ) -> anyhow::Result {
|  ____________________________^
71 | |         self.fetch_manifest_digest(image, auth).await
72 | |     }
| |^ future created by async block is not Send
|
= help: the trait Sync is not implemented for core::fmt::Opaque
= note: required for the cast to the object type dyn futures::Future<Output = Result<std::string::String, anyhow::Error>> + std::marker::Send
error: could not compile kubelet due to 2 previous errors

On arm64 SBC, we can do as follows.
sudo apt install musl-tools
cd /usr/bin
sudo ln -s musl-gcc aarch64-linux-musl-gcc
Cargo.toml file: [dependencies]
openssl = { version = "*", features = ["vendored"] }
rustup target add aarch64-unknown-linux-musl
cargo build --target aarch64-unknown-linux-musl --release


output branchRollback failed log

Ⅰ. Describe what this PR did
Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)?
Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews

Codecov Report

Merging #1201 into develop will decrease coverage by 0.01%.
The diff coverage is 0%.

@@              Coverage Diff              @@
##             develop    #1201      +/-   ##
=============================================
- Coverage      43.48%   43.46%   -0.02%
  Complexity      1427     1427
=============================================
  Files            250      250
  Lines          10220    10222       +2
  Branches        1330     1331       +1
=============================================
- Hits            4444     4443       -1
- Misses          5190     5192       +2
- Partials         586      587       +1

Impacted Files
Coverage Δ
Complexity Δ

...java/io/seata/rm/datasource/DataSourceManager.java
27.41% <0%> (-0.92%)
7 <0> (ø)

...server/store/file/FileTransactionStoreManager.java
45.99% <0%> (-0.35%)
19% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fae1276...2cadfe1. Read the comment docs.


AuthenticatedClient overhaul, docstrings, and testing

This is an overhaul of the AuthenticatedClient. A summary:

Consistent interface for all endpoints (#60)
Paginated requests have been abstracted away in the form of generators (#64)
Docstrings for all methods (#56)
Tests for as many authenticated endpoints as possible. Sandbox API credentials need to be supplied to run the tests (instructions in readme). The sandbox API is a little flaky, so some tests randomly fail. Not the best testing methodology, but it exercises most of the functions at least.

An AuthenticatedClient object now makes all HTTP calls through a single requests.Session. This may give a performance boost when making many calls to the GDAX API and should remain thead-safe. All HTTP requests are made through helper functions for cleaner code, and should also make implementing error handling simple to do in the future. These changes are implemented in PublicClient as well.
This PR has breaking changes in the interface, so we can discuss how to integrate without losing sync between Github and pypi again. Maybe push this to a version branch, and then merge to master when ready to update pypi?
Thanks to @Jigsaw-Jams for writing most of the docstrings!

Looks like a pretty solid PR. I'll try to find some time to test it and make comments if you'd like some feedback.

@PaulMest That would be good if someone else could try out the new tests! I never tried running them on a fresh GDAX sandbox account, so that would be a good check.

I like all of this, with the exception of refactoring orders.  While I agree we may add some more user friendly "stop_limit_order" methods, we should not remove buy or sell.  I almost feel like we should replace them all with "place_order" that other order types may be built on.   My goal is to keep this wrapper as true to the official GDAX API doc's as possible.  Does this make sense?  Please let me know your thoughts.

@danpaquin I'm not sure I fully followed you - would adding buy and sell as wrappers of place_order work for you?
Right now place_order is the main function with all the logic, and place_market_order, place_limit_order, and place_stop_order simply call it. My motivation in splitting the order function up was to make the code more user-friendly. Each order type has a large number of parameters, so having separate functions with named parameters and corresponding docstrings should hopefully make it easier to build a proper order without having to consult the GDAX API spec. I definitely agree with trying to stick to the spec - that was my intent!

Can we at least merge the readme update soon? It says you can use product_id when initializing AuthenticatedClient but it fails on the pip version and the GitHub version.

It would be nice to have the same type of pagination in public_client as well. Like for get_product_trades(), for example. Maybe  _send_paginated_message() can be shared between the two?

I love the pagination, docstrings and all other improvements, but I would not like to break other backwards compatibility.  We can keep as it, but add back buy() and sell() methods for backwards compatibility.
Thanks!

@dmitrytorba Thanks for the review and suggestion, I implemented it in latest commit.
@danpaquin Thanks for the comment - I added back buy() and sell(), they should be backwards compatible now.
Now both PublicClient and AuthenticatedClient use  _send_message and _send_paginated_message for every API call. Clean and consistent across the whole library.

@acontry, awesome work!
looks like there are conflicts with master

@dmitrytorba Thanks! Updated this branch one more time...

I am nervous to implement such drastic changes in one PR.  Backwards comparability is my main concern regarding order's.  I would like to keep the wrapper as close to bare-bones as possible so users can look at the wrapper README alongside the GDAX API Reference.  Higher level methods can be added in developers' inherited classes.

@danpaquin Backwards compatibility is there for orders. I understand this is a big PR now - if these changes are still desired, I would suggest creating a dev branch where these changes can be merged. That makes it easier for them to be fully vetted before merging into master.

Just a lurker passing by: I can see the awesome work done on this module in general and in particular in this PR.
From a newbie point of view, the documentation added to AuthenticatedClient is great!
I hope you can merge it soon with the master so that I can develop with it.

Hi Bkyn, thank you for sharing your feedback.  I will not merge to master since many methods have been converted to generators which may break older code.  I love the docstrings and refactors, but this PR has become too big of a change to push all at once. I will work on creating a dev branch to pull this into.

@danpaquin sounds good to me! I just pointed this PR at development branch for you.


simpler way to ignore autorepeats

-check for event.repeat (not supported in Safari currently)
-full arrow key movement

Excelent! i wasnt aware of event.repeat, cool to learn.
i love your tron demo, the sounds are real cool :)


Determine which serialisation / compression algorithm to use throughout the pipeline

In order to reduce our data storage requirements, we should make use of compression as long as the benefits of doing so exceed any associated costs. To this end, we need to select a compression algorithm which weighs both the associated compression speed, and compression ratio.
To consider this issue complete, we need to identify:

[ ] A compression algorithm to be used when submitting data to the collectors from a given ooni-probe;
[ ] A compression algorithm to use throughout the pipeline when dealing with intermediary files; and
[ ] A compression algorithm to use for long-term archiving

In certain cases, we may find the use of compression to be unnecessary (e.g. when dealing with intermediary I/O at each stage of the pipeline).
The top contenders right now are:

pickle + lz4
msgpack + zlib

I'll perform some A/B tests, and include my findings here.
@joelanders @hellais @darkk @willscott

pickle

I'd rather avoid storing anything in a format tightly coupled with specific language unless it gives us major performance boost. It may be a good baseline, but I'm a bit sceptical about usage of pickle for data exchange in production.

@darkk I agree wholeheartedly. pickle/dill would only really be advantageous when working with metrics which haven't been normalised, and may not necessarily be JSON / BSON / msgpack serialisable.

I ran some benchmarks on the ooni-datacollector machine with the goal of identifying a good serialisation format for long term storage.
https://gist.github.com/hellais/c3914475331a313266c3f51e49bd287d
These results seem to indicate that lz4 could be a good option.

There are some more notes on compression and serialization I've gathered while passing some smoke through the pipeline.
tl;dr: tar small files — save IOPS, compress tarred — save bandwidth & space, read blocks not lines — tarfile line-by-line reading code is slow.
First, LZ4 does not currently implement checksums for compressed data (aka Block checksum), only content checksum is implemented, so we have to add some checksum that can be used for data scrubbing. I suggest to use crc32 due to following reasons:

IMHO speedy checksum is better for scrubbing, I'm checksumming raw files with SHA1 and CRC32 to make scrubbing IO-bound instead of CPU-bound on every platform we can imagine
the checksum should be easy to verify for others – xxhash, murmur cityhash don't have wide-spread CLI tools
crc32 is not "strong", but lz4 also has xxhash content checksum, so I don't consider checksum "strength" as a requirement
sha1 is a nice fallback for crc32, computation costs nothing as "canning" is bottlenecked by disk write

box
disk
crc32
sha512
sha256
sha1
md5

darkk-laptop i7-6600U, SSD
450
1120
576
317
846
568

datacollector E5645, HDD? ceph?
82
830
251
164
459
527

ssdamsterdam  E5-2620, SSD
512
550
167
113
359
343

crc32 performance was estimated with python2.7 -m timeit --number 100 --repeat 10 --setup  'import zlib; a = "A"*(1024*1024*16)' -- 'zlib.crc32(a)', others — with openssl speed.
Second, python-lz4 does not implement streaming decompressor yet. That's not a big deal, we can use CLI lz4 tool when we have to stream whole file. Sanitized files have small frames to be seekable, so streaming decompressor is required only for reading raw (canned) files.
Third, lz4 is byte-oriented and not bit-oriented, so it does not compress random base64-encoded strings well, it means, that we MAY want to have different compression codes for different reports if/when amount of binary base64-encoded data rises (pcaps, TLS certificates, non-unicode text of the webpages, and so on).
$ cat urandom.1Mb | base64 | wc --bytes
1416501
$ cat urandom.1Mb | base64 | gzip -6 | wc --bytes
1076437
$ cat urandom.1Mb | base64 | lz4 -6 | wc --bytes
1416520

Fourth, legendary slowness of yaml processing has two causes, yaml library is not the only one. yaml.loader.CLoader processes ~40 Mbytes of raw yamly reports per second, BUT reading tons of tiny yaml files with simple cat *.yaml is way slower than 'usual' disk speed, it was less than 10 Mbyte/s on CEPH-based disk. Compressing several yaml files into tar archive and streaming it to the CLoader works significantly better being bottlenecked by CPU. Mobile app will produce alike pattern as it was going to have quite small reports.
tarfile itself does not have major overhead, stream decoding speed is ~450 Mbyte/s at datacollector and ~530 at darkk-laptop (given cached tarfile as an input). IO-related measurements were done with vm.drop_caches=3 to ensure that the data is not read from local pagecache.

I believe we have now settled on using a tar container compressed using lz4, so I think this can now be closed.
@darkk am I correct?

Yep.
And .tar.{gz|bz2|xz} for cold storage in glacier as lz4 is not settled yet -- TheTorProject/ooni-pipeline#57
And adding gzip to collector , ooni-probe and MK (zlib is part of iOS SDK and Android NDK) as every report is POSTed line-by-line, uncompressed (probably, that deserves some batching and/or keepalive) -- TheTorProject/ooni-backend#106.


Clean up the HighlevelGraph.dicts property

HighLevelGraph.dicts is an alias property to .layers, which was put there as temporary backwards compatibility when we migrated from the previous design in #4092.
There's a bunch of stuff in the dask codebase that still invokes .dicts, but it is trivial to fix; not sure if anything else in third party projects uses it though. Should we go through a formal deprecation process, or is it OK to just remove it as over 2 years have passed since the change now?

Thanks for bringing this up @crusaderky. I'm not sure how much existing code out there is using .dicts. It's doesn't seem too expensive from a maintenance perspective to update all the code here to use .layers and add a deprecation warning to .dicts for others to do the same.

Thanks for bringing this up @crusaderky. I'm not sure how much existing code out there is using .dicts. It's doesn't seem too expensive from a maintenance perspective to update all the code here to use .layers and add a deprecation warning to .dicts for others to do the same.


Makefile: handle cases where $GOPATH is a list

In multiple places, we've been assuming that we can invoke binaries that we install as $GOPATH/bin/$binary.  This doesn't work in cases where $GOPATH is a list.

:+1: I will test this out.

LGTM
cc: @sameo @runcom PTAL

@jawnsy Want to take a look?

LGTM


Use the atom-text-editor tag instead of the editor class.

In keymaps/rubymotion.cson: Use the atom-text-editor tag instead of the editor class.

:+1:


Use real data for test

We should have this pub follow some existing pubs so that we have real data to test with.

[ ] should follow an existing pub
[x] need to use the main caps file for ssb
[x] We want to have one with test data also, need a separate env for this. Depends on #14
[x] need to paginate response #7

You might need to increase the storage space on the server

See https://github.com/ssb-ngi-pointer/jitdb/issues/188 -- paginate is not working. We're using too much memory and crashing


fix: Fixed content-type check in default_verifier

Pull Request Checklist

[X] Make sure to read the contributor guide here.
[X] Make sure you are making a pull request against the main branch.
[X] Make sure your pull request title matches the requested format.
[X] Make sure to lint, type-check, and run unit and API tests.

Description
Fixed issue with content-type detection in default_verifier function.

:tada: This PR is included in version 4.0.4 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Support for Dameng user and role

Support for Dameng user info and role info.
https://github.com/dbeaver/dbeaver/pull/22391

Thanks again for your contribution @Stkai!
Changes will be available in the next release on 22-01-2024.


TurboHookV2 which supports discord's embeds

It tested it beforehand to make sure it all works correctly.
It sends embeds data like title, timestamp, color, images, links, thumbnail, footer, and description.

There's no need for this PR. I made a PR that already did this, but I closed it because I decided the current version of TurboHook is too messy have a rewritten version that loads better that I plan to PR soon.

This was the update: https://github.com/TurboWarp/extensions/pull/706
I closed it because I rewrote the entire extension from the ground up, that makes it a fully capable normal multi purpose WebHook extension that'll have a lot of useful functionality. So I request this be closed, I don't need the bad code to be expanded on.

This was the update: #706 I closed it because I rewrote the entire extension from the ground up, that makes it a fully capable normal multi purpose WebHook extension that'll have a lot of useful functionality. So I request this be closed, I don't want the bad code to be expanded on.

ok cool, thanks, also how the hell do I use prettier 💀💀💀😭 also where is the better version

@fcures you have to either have to install the deck server and command line use prettier using npm run format, or use the prettier VSCode extension

The better version still needs one more thing till release

The better version still needs one more thing till release

Alright cool


Add Crossformer as an imputation model

What does this PR do?

add Crossformer as an imputation model, as suggested in #295;

Before submitting

[x] Was this discussed/approved via a GitHub issue? Please add a link to it if that's the case.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have written necessary tests and already run them locally.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

205 of 246 (83.33%)  changed or added relevant lines in 7 files are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.03%) to 82.882%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pypots/imputation/crossformer/model.py
68
72
94.44%

pypots/imputation/crossformer/modules/submodules.py
90
127
70.87%

Files with Coverage Reduction
New Missed Lines
%

pypots/cli/env.py
2
61.22%

Totals

Change from base Build<PHONE_NUMBER>:
0.03%

Covered Lines:
6270

Relevant Lines:
7565

💛  - Coveralls


Bump to EAI v1.12.x

To address this issue, EAI needs a bump. Hoping this can get in before the full v2 bump, for those of us still in the middle of addon transitions. :sweat_smile:

(Effectively a duplicate of #913)

Closed by https://github.com/emberjs/ember-qunit/pull/921.


Is it possible to add charts with animation

Hi Rossning92，
This is a great framework, I’m not sure if my idea can be realized, Is it possible to add charts with animation.
I hope to get your advice or help.
Best Regards
Kevin

Hey, @evenkiss, just added an example to create line chart animation. Please refer to /examples/line-chart.js.


internal error

go version: go1.20.6
internal error: package "encoding/json" without types was imported from xxxx

duplicated to #2


Jenkins: fix Docker image build on Linux for master branch

Description of the Change
Docker build could have failed when Dockerfile was changed and committed to master branch because the new Dockerfile was compared to itself instead of previous version.
https://jenkins.soramitsu.co.jp/job/iroha/job/iroha-hyperledger/job/master/218/consoleText
Benefits
Successful master build on Dockerfile changes
Possible Drawbacks
None

Mater is build and push every time, referenceBranchOrCommit is used just to decide should we use docker cache while building.

@BulatSaif Yes, and cache is used incorrectly when triggered by a commit in master branch.