Tango routing broken?

I'm currently preparing the next run of the the Go HTTP Routing Benchmark. Since several broken implementations were discovered already, I added a test validating the routing of each router earlier: julienschmidt/go-http-routing-benchmark@b5a1ebab84d686b0675a3c0c7d0b928a3b998d02
Unfortunately it seems like there is a problem with Tango:
	routers_test.go:78: Tango in API GitHub: 404 - Not Found; expected GET /notifications/threads/:id
	routers_test.go:78: Tango in API GitHub: 404 - Not Found; expected GET /gists/:id
	routers_test.go:78: Tango in API GitHub: 404 - Not Found; expected DELETE /gists/:id
	routers_test.go:78: Tango in API GitHub: 404 - Not Found; expected GET /repos/:owner/:repo/issues/:number
	routers_test.go:78: Tango in API GitHub: 404 - Not Found; expected GET /repos/:owner/:repo/milestones/:number
	routers_test.go:78: Tango in API GitHub: 404 - Not Found; expected GET /orgs/:org
	routers_test.go:78: Tango in API GitHub: 404 - Not Found; expected GET /teams/:id
	routers_test.go:78: Tango in API GitHub: 404 - Not Found; expected DELETE /teams/:id
	routers_test.go:78: Tango in API GitHub: 404 - Not Found; expected GET /repos/:owner/:repo/pulls/:number
	routers_test.go:78: Tango in API GitHub: 404 - Not Found; expected GET /repos/:owner/:repo
	routers_test.go:78: Tango in API GitHub: 404 - Not Found; expected DELETE /repos/:owner/:repo
	routers_test.go:78: Tango in API GitHub: 404 - Not Found; expected GET /repos/:owner/:repo/commits/:sha
	routers_test.go:78: Tango in API GitHub: 404 - Not Found; expected GET /repos/:owner/:repo/releases/:id
	routers_test.go:78: Tango in API GitHub: 404 - Not Found; expected GET /users/:user
	routers_test.go:78: Tango in API GPlus: 404 - Not Found; expected GET /people/:userId
	routers_test.go:78: Tango in API GPlus: 404 - Not Found; expected GET /activities/:activityId
	routers_test.go:78: Tango in API Parse: 404 - Not Found; expected GET /1/classes/:className

I need your help to figure out what is broken and to fix it.
The current implementation for Tango is here: Tango Implementation
The complete list of routes can be found here: GitHub, GPlus, Parse
Thanks in advance!

I will check this. Thanks.

Please go get -u github.com/lunny/tango and accept github.com/julienschmidt/go-http-routing-benchmark#29. I have tested. There is no error.

That is embarrassing.. the go get -u github.com/lunny/tango indeed fixed it. I thought i just cloned it. Sorry for bothering you!


Export code not working

When running the exported code, it outputs the error msg. This is likely the side effect of updating python API.

Yeah jupyter_bifrost_tracing was never updated to track chart.plot() instead of df.bifrost.plot()`

Fixed this!


CausalELM v0.2.1

@JuliaRegistrator register()

Error while trying to register: "Tag with name v0.2.1 already exists and points to a different commit"


Failed to connect to server

Azure Data Studio Version:   1.4.5, on Windows 10

Steps to Reproduce:

add managed connections to several Azure SQL server(s) with "default" db.
double click to "connect" and it connects
when you click open Databases below it, it shows: "Failed to connect to server".
NOTE:  this doesn't always happen... it happens only sometimes.

With many recent improvements, this should be resolved now.
Let us know if you encounter this again.


Markdown Heading Editor Folding Not Working

VSCode Version:1.49.1
OS Version:Ubuntu 20.04.1

Steps to Reproduce:

The markdown folding as illustrated in [folding] Add code folding for markdown based on heading level · Issue #3347 · microsoft/vscode, doesn't work. I don't get the option to fold it

Does this issue occur when all extensions are disabled?:  Yes

Please markdown code that demonstrates the problem

@mjbvz
I opened the following markdown file after disabling all extensions

Observation
We can see that we don't get the V symbol for headings, so we are  unable to minimize it.
Reference
Version: 1.49.1
Commit: 58bb7b2331731bf72587010e943852e13e6fd3cf
Date: 2020-09-16T23:23:33.049Z
Electron: 9.2.1
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: <IP_ADDRESS>-electron.0
OS: Linux x64 5.4.0-48-generic

Please share the text of that file

---
title: 'Sewing'
subtitle:
author: 'Anish Mittal'
date: 'Wednesday, October 07, 2020 - 18:23'
keywords: ['']
ratings: ['']
reviews: ''
---

# Introduction

- [How to Sew by Hand - YouTube](https://www.youtube.com/watch?v=ZvzMMcKHVR4)
- [Hand sewing 2 Sewing on a button - YouTube](https://www.youtube.com/watch?v=MjmUaWkzF-I)
  - Excellent!
- [How to Sew a Zipper, TWO ways - YouTube](https://www.youtube.com/watch?v=oHDDhUzIXPY)
  - Nice! But for special use.

## Summary

-

# Bibliography

-

---

That example works fine for me:

Did you disable the built-in markdown language features extension?

Or actually, either you've disabled the built-in extension or have changed the "editor.foldingStrategy" setting to be indentation based instead of smart

Interestingly I'm having a very similar problem using Org Mode after just updating VSCode. It no longer recognizes the asterisks as the headings that should be folded, and will only fold lines that start with a dash like the OP showed here.
Version: 1.50.1
Commit: d2e414d9e4239a252d1ab117bd7067f125afd80a
Date: 2020-10-13T14:53:05.704Z (1 wk ago)
Electron: 9.2.1
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: <IP_ADDRESS>-electron.0
OS: Darwin x64 18.0.0

I struggled with this too for a while, even after checking the "Editor:Folding" setting, which was mysteriously off by default.  But it appears that some folding arrows only appear when you mouse over the gutter (like Markdown), but other arrows are persistent (like list-level folding).


OUTCAR parser produces more errors when reused

I added a test-case to branch multiple-runs. This test case parses two outcars with the same parse. It produces errors that are not produced, when different parser instances are run on the same outcars. The error is Inconsistent number of ions and species..
This might be related to this forum entry: https://matsci.org/t/wrong-parsing-in-oasis-possibly-some-bad-settings-somewhere/36150

@ladinesa please have a look at this
@ondracka F.Y.I


Make transforms.functional_tensor functions differential w.r.t. their parameters

Linked Issue: #5000
Make operations in torchvision.transforms.functional_tensor differential w.r.t. hyper-parameters, which is helpful for Faster AutoAugment search (hyper-parameters are learnable parameters via backward).
Some operations are not differential (e.g., Posterize), which might require users to write their own implementations.
Todo List:

[ ] JIT support
[x] Affine has no backward gradient w.r.t. matrix.
[x] Brightness, Contrast, Hue, Saturation, Gamma, Sharpness, Solarize
[x] Affine, Rotate (require some modification in torchvision.transforms.functional)
[x] corresponding annotation of methods in torchvision.transforms.functional
[x] (solved in #4999 )autocontrast seems to be incorrect. Need further check.

https://github.com/pytorch/vision/blob/18cf5ab1c1307e9d0e1b1e4d26ce9c45e04f8376/torchvision/transforms/functional_tensor.py#L935-L945
The input Image could be either (N, C, H, W) or (C, H, W), which makes L940 to rely on different dimensions. This is inconsistent behavior for these 2 cases.
Other implementations as reference:
https://github.com/moskomule/dda/blob/master/dda/functional.py#L203
https://pillow.readthedocs.io/en/stable/_modules/PIL/ImageOps.html#autocontrast

If any maintainer thinks this worth doing, I'll continue to work on it.

@ain-soph Thanks for opening this.
What you propose is interesting but it might be worth discussing the API prior writing the PR. This is to avoid a situation where you spend lots of time working on the feature and not being able to merge due to some unforeseen limitation (JIT might give us headaches for example). Since you already opened a PR, we can discuss this here as well if you prefer.
Concerning the autoaugment behaviour, I would like to understand more why you say the behaviour is inconsistent.

The input Image could be either (N, C, H, W) or (C, H, W), which makes L940 to rely on different dimensions. This is inconsistent behavior for these 2 cases.

My understanding is that the lines at question always look for the min/max across H and W no matter the input. The next 2 lines ensure that we won't divide by 0. Could you provide an example that show-cases the issue?

@datumbox
import torch
import torchvision.transforms.functional as F

a=torch.rand(1,3,32,32)
a=a/2+0.3

(F.autocontrast(a)[0]==F.autocontrast(a[0])).all()   # True

a[0,2]=0.7

(F.autocontrast(a)[0]==F.autocontrast(a[0])).all()   # False

We should expect it to be True. Because (1,C,H,W) should be same as (C,H,W). But the [0] in L940 makes it rely on the first dimension of the image, which might be N or C.

@ain-soph Thanks a lot for reporting and for submitting a snippet that reproduces the issue. I submitted an alternative solution for this at #4999. Let me know your thoughts.

@datumbox
Almost done (except JIT, which I'm totally unfamiliar). But there is one issue about the Affine.
The current implementation seems to have no gradient w.r.t. matrix (all zero).
https://github.com/pytorch/vision/blob/47281bbf9b8cd7be952ec23c69c0f62e4733c2a2/torchvision/transforms/functional_tensor.py#L694-L704
I have debugged and find grid.sum().backward() will have gradient w.r.t. matrix, but L704 loses the gradient information, which means _apply_grid_transform(img, grid, interpolation, fill=fill) has no gradient w.r.t. grid.
However, it relies on torch.nn.functional.grid_sample, which seems to be a C++ implementation. Any further thoughts on how to make it differential?
I tried kornia.geometry.transform.affine (link) and it works well.

@ain-soph I agree with @datumbox that prior to the PR it would be better to open a feature request issue and give motivations about the requested feature.
According to the code in the PR, your idea is to propagate grad through the transformations and make sure that arguments could be of type nn.Parameters or tensors with grads. I think if we could make it without any BC breaking changes, it would be nice otherwise we have to argue why it is more important to keep gradients and make args learnable vs previously working API.
As for grid_sample op and grad propagation for grid, this is done by pytorch. For example:
import torch
from torch.nn.functional import grid_sample

img = torch.rand(2, 3, 4, 4, requires_grad=True)
grid = torch.rand(2, 6, 6, 2, requires_grad=True)

out = grid_sample(img, grid, align_corners=False)
assert out.requires_grad
out.mean().backward()
img.grad is not None, grid.grad is not None
> True, True

@vfdev-5 Thanks for your response!
You are right, grid_sample has backward support. I've tested more cases and find the reason why it doesn't work here is that mode="nearest". If I change it to bilinear everything works.
Don't know if it is the expected behavior to be all zero gradient when mode="nearest".
I'll later propose a feature request issue linked to this PR.

I think the rotate function seems to be unnecessary since it's already covered by affine.
I think it might be a better option to deprecate rotate and arrange default values for arguments (e.g., angle=0.0, shear=[0.0, 0.0], transpose=[0.0, 0.0]) in affine, so that we only need to call affine(tensor, shear=shear) without setting other arguments.

I think the rotate function seems to be unnecessary since it's already covered by affine. It might be a better option to deprecate rotate to make API more unified. But it's totally fine to keep current structure.

@ain-soph due to historical reasons it was first implemented rotate and then affine ops. Rotate API is not completely redundant compared to affine op, as rotate has flag expand which is not present for affine op.
I do not think that deprecating this op in favor of affine is a good idea now. Replacing the implementation of rotate and use affine behind can be checked, especially in terms of performances on PIL and tensors (it may appear that rotate can be a bit faster then affine with angle only)

It would be helpful if there are default values for arguments (e.g., angle=0.0, shear=[0.0, 0.0], transpose=[0.0, 0.0]) in affine, so that we only need to call affine(tensor, shear=shear) without setting other arguments.

Yes, I think default values could be helpful in this case.

Not sure if it is the expected behavior to be all zero gradient when mode="nearest". It seems correct to me that nearest sampling makes the gradient irrelevant to grid.

yes, it is expected (https://github.com/pytorch/pytorch/blob/ca945d989ac56e0070f82fbe657d078610043ea3/aten/src/ATen/native/cpu/GridSamplerKernel.cpp#L806). Also, take a look at section 3.3 "Differentiable Image Sampling" of this paper where they derive forward pass formulas for nearest and bilinear modes and how to compute grid gradients for bilinear case.

Maybe we should modify the default interpolation mode to be bilinear?

I think we should not change default value to cover particular case of non-zero grid grads. For you use-case you can specify interpolation mode and thus obtain will obtain non-zero grads.

@vfdev-5 So you preference is that the default argument value of affine is still interpolation='nearest'?
If so, I'll modify the codes.
And any idea on how to fix this issue?
        center_f[0] = 1.0 * (center[0] - img_size[0] * 0.5)
        center_f[1] = 1.0 * (center[1] - img_size[1] * 0.5)

Error Log:
Traceback (most recent call last):
  File "/root/project/test/test_functional_tensor.py", line 62, in <module>
    class TestRotate:
  File "/root/project/test/test_functional_tensor.py", line 65, in TestRotate
    scripted_rotate = torch.jit.script(F.rotate)
  File "/root/project/env/lib/python3.7/site-packages/torch/jit/_script.py", line 1312, in script
    qualified_name, ast, _rcb, get_default_args(obj)
RuntimeError:
'Union[Tensor, List[int]]' object is not subscriptable:
  File "/root/project/torchvision/transforms/functional.py", line 1081
        img_size = get_image_size(img)
        # Center values should be in pixel coordinates but translated such that (0, 0) corresponds to image center.
        center_f[0] = 1.0 * (center[0] - img_size[0] * 0.5)
                             ~~~~~~~~~ <--- HERE
        center_f[1] = 1.0 * (center[1] - img_size[1] * 0.5)

The center_f is Union[Tensor, List[int]] right now.

@ain-soph checkout https://github.com/vfdev-5/vision/commit/1214e6f1256f3bb403c0a1b29464e2c6e7d82859
It still does not pass JIT checks for center as tuple of ints but could be tweaked later...

@ain-soph checkout vfdev-5@1214e6f It still does not pass JIT checks for center as tuple of ints but could be tweaked later...

So your preference is that the default argument value of affine is still interpolation='nearest'?

yes, let's make things less BC-breaking possible: no public API breaks, but only private methods changes

I believe it is a typo in the main branch and center type should be List[float] rather than List[int].
I'm also working on the JIT issue. Hope I can fix it tonight.

@ain-soph let's make the following to avoid debugging on the CI, I'll finish up the code for rotate that will pass JIT etc in another PR and you could then make other color-related transforms and affine. What do you think ?

@vfdev-5 Oops, sorry if it causes any trouble when I debug...
I try to debug locally, but it seems I have to do some extra work from source code to final binary version.
I'll appreciate if you can help me with JIT issues. I have no experience with JIT previously. I'm still facing a ton of errors, even after I have fixed a bunch of them.

@vfdev-5 I've get my local environment ready for debugging, and it seems all tests have passed except the new differentiable test that you added.
Here are the only 2 issues remaining:

(personal guess) the current type of angle declared in rotate is float, which makes the Tensor converted to float in JIT script function. Therefore, the gradient information is lost.
the current type of center declared in rotate is List[float], but the test example passes Tensor

And another CI error that I can't reproduce at local environment:
test_random_rotate[85-InterpolationMode.NEAREST-degrees3-True-center0-cuda] - test.test_transforms_tensor

Traceback (most recent call last):
  File "C:\Users\circleci\project\test\test_transforms_tensor.py", line 555, in test_random_rotate
    _test_transform_vs_scripted(transform, s_transform, tensor)
  File "C:\Users\circleci\project\test\test_transforms_tensor.py", line 29, in _test_transform_vs_scripted
    assert_equal(out1, out2, msg=msg)
  File "C:\Users\circleci\project\env\lib\site-packages\torch\testing\_comparison.py", line 1217, in assert_close
    assert_equal(
  File "C:\Users\circleci\project\env\lib\site-packages\torch\testing\_comparison.py", line 997, in assert_equal
    raise error_metas[0].to_error()
AssertionError: Tensor-likes are not close!

Mismatched elements: 3 / 9333 (0.0%)
Greatest absolute difference: 43 at index (0, 3, 2) (up to 1e-06 allowed)
Greatest relative difference: 0.5058823823928833 at index (0, 3, 2) (up to 0 allowed)

close in favor of #5110


Nested attribute selectors

6ae9938/2.7.0 introduced a bug where nested attribute selectors (div[class="foo"] div[class="bar"]) aren't transformed to XPath properly anymore. Does anyone have an idea how we could fix this?

I've added a commit with an attempt to fix the problem. I don't know if it's ideal though.

I've updated the PR with your first suggestion. Thank you for your help!

Thanks, @MidnightDesign! When merging, I added more tests, and included changes to ensure we properly identify quoted strings based opening quote.


refactor(transformer/common): VarDeclarations insert either var or let statements

Add ability for VarDeclarations to insert let declarations as well as var declarations. This is required for class properties transform.
Implementation note: var and let declarators are stored in 2 separate ArenaVecs. This allows using those ArenaVec<Declarator<'a>>s directly in the AST, rather storing all Declarators in a single Vec and having to loop through it when inserting the var / let statements to split it into 2 ArenaVecs of var / let declarators. I'm not completely sure this is better than using a single Vec for both, but I think probably it is.

#7043  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @overlookmotel and the rest of your teammates on  Graphite

Merge activity

Nov 1, 12:08 AM EDT: The merge label '0-merge' was detected. This PR will be added to the Graphite merge queue once it meets the requirements.


Unknown word error

The following gulp task throws Error: postcss-style-guide: <css input>:1:1: Unknown word output.css
gulp.task('guide', function() {
  var processedCSS = 'output.css';
  return gulp.src('src/*.css')
    .pipe(require('gulp-postcss')([
      require('postcss-style-guide')(processedCSS, {
        name: "Project Name"
       })
    ]))
    .pipe(gulp.dest('build/'));
});

The css is taken directly from the gulp example in the readme. Any idea what is causing the error?

@jdozierezell Sorry, my bad in README. Please change var processedCSS = 'output.css'; to var processedCSS = fs.readFileSync('output.css', 'utf-8');.

Thanks. That was it.


Date and Time

How to date and time?
DD/MM/YYYY hh:ii:ss
What format?

Hi @thiagoreri, please check out the Moment.js documentation about all token formats at http://momentjs.com/docs/#/displaying/format/. In your case, you just need to change the format to DD/MM/YYYY hh:mm:ss.


x/tools/go/gcimporter15: backport changes from std lib version

gcimporter15 needs to be updated to incorporate the latest changes to std lib gcimporter.

Is this done? I thought I saw some similar-sounding CLs fly by.

@bradfitz Indeed. Forgot to mention https://go-review.googlesource.com/18772,  https://go-review.googlesource.com/18771, and https://go-review.googlesource.com/18770 which together fixed this issue.


Crash when opening menu during input or confirmation dialog

Issue
If the Caffeinator menu is opened after the pre-execution prompt or an input dialog is opened, the app crashes as soon as the dialog is closed (this does not affect error dialogs or the Check for Update dialog). It has been confirmed that runModalInFront is always being called on the main thread.
Steps to Reproduce

Start Caffeinator in timed mode/Caffeinate a Process/start Caffeinator with Prompt Before Executing enabled (generate one of the relevant pop-up dialogs)
Re-open, then close, the Caffeinator menu while the dialog is open
Close out of the dialog (Cancel/OK)
App crashes

This issue was initially addressed in 307d1da6da26609716a648b9f7d297a966f273b3, but continues to occur if a window (e.g., the Help window or the Caffeinate with Args window) is open when the menu is activated.

This issue was incorrectly diagnosed—the actual issue lies in the pre-execution dialog specifically; other dialogs are not affected and were properly fixed in the aforementioned commit. Now resolved as of e63bd92.


Test/join-page-button

testflow after clicking join button

logged in user  -> join-page
not logged in -> join-page -> alert -> redirect to signin

Hey @khushi818, There's an extra if statement at line 34 in test file. Also, for better readability, you can use a switch statement and perhaps refactor lines that are redundant.
Something like :
if (response.url().endsWith('/users/self') && page.url().endsWith('/join')) {
	switch (response.status()) {
		case 401:
			delay(2000);
			confirmAlerts();
			break;
		case 200:
			browser.close();
			break;
	}

Oh thanks for pointing out. I didn't notice.
and yes it is a good suggestion. will change it right away

@khushi818 LGTM

testflow after clicking join button

logged in user  -> join-page
not logged in -> join-page -> alert -> redirect to signin (join-button.test.js)
incomplete details - redirect back to home page (join-buton-signup.test.js)

Hey @khushi818 you mean to this:- not logged in -> join-page -> alert -> redirect to signin
not sign-in with github-> join-page -> alert ->redirect to sign-in with github??
incomplete details - redirect back to home page
its means incomplete details of join form?

testflow after clicking join button

logged in user  -> join-page
not logged in -> join-page -> alert -> redirect to signin (join-button.test.js)
incomplete details - redirect back to home page (join-buton-signup.test.js)

Hey @khushi818 meaning of this line in description:- not logged in -> join-page -> alert -> redirect to signin

not sign-in with github-> join-page -> alert ->redirect to sign-in with github??

incomplete details - redirect back to home page

its means incomplete details of join form?

incomplete details means the they are not signed up through signup page and logged in user means they have all the details completed


Power Query Table.Group - GroupKind options no longer documented

Link to article:
Power Query Table.Group function
Problem:
Prior to moving the documentation to docs.microsoft.com the Table.Group GroupKind options were documented, they're no longer:

GroupKind.Global has no explaination of what this option is/means
GroupKind.Local has no explaination of what this option is/means

From Power Query M Reference PDF:

A group can be local (GroupKind.Local) or global (GroupKind.Global)
A local group is formed from a consecutivesequence of rows from an input table with thesamekey value
A global group is formed from all rows in an input table with the same key value
Multiple local groups may be produced with the same key value but only a single global group is produced for a given key value
The default groupKind value is GroupKind.Global

I'd also suggest to check and test the feature. I've found issues when you join a table result of a Table.Group with another table

@LazyLorenzo Please move this issue to https://github.com/MicrosoftDocs/powerquery-docs/issues

@rootsmusic Why not. Problem is that I don't see how to do it :(

@LazyLorenzo You can open an issue in https://github.com/MicrosoftDocs/powerquery-docs/issues and refer to this one by number.


WHATWG-compliant?

Does this parser attempt to follow the spec that browsers follow?
Rust crates that follow WHATWG (I think this list is complete):

lol-html
html5ever
html5gum

In its benchmarking suite, tl compares itself against both kinds of parsers, ones that do attempt to comply to the WHATWG spec and parsers who don't. Since WHATWG defines error-recovery etc very precisely, that influences what kind of optimizations one can do, and explains why html5ever is slow.

thanks for bringing this up, it's a good point and I think it's important to add this to the README. Seems a little unfair to make crates like html5ever look "bad" in the benchmarks even though the reason for that is probably that they closely follow the spec as you say, and comparing it to this crate. This is mentioned in the separate benchmark repo (linked in the README), but it's kind of hidden behind a wall of text (and not mentioned here), which is unfortunate.
Currently, this crate doesn't follow the full spec. When I made this crate, I needed a fast library for a project, something that can parse "sane" HTML documents very quickly (doesn't need to be spec compliant) and provides a simple API to interact with the parsed tree. Hopefully in the future we can work towards being whatwg-compliant, without losing too much performance. Also html5gum looks cool ;)

I added a few things to the README, hoping that it makes the goals of this crate more clear. I've also fixed the benchmarks section up a bit. The table now has a column for "follows spec" (whether compliance with the spec is a goal) and a "note", saying that it's important to understand what difference it makes for performance if one isn't bound to a specification.


Update for new cashscript release

Hey, sorry for not getting to this earlier. I started testing this today and everything looks good so far! Will merge and update the package soon!

Thanks for your PR! Everything looks good. I was unsure about your grammar in CashScript.g4. I see you've changed the syntax for the requireStatement and added syntax for the consoleStatement. I'm assuming this is in preparation for this upcoming release so I will merge it in shortly.

Hey thanks for checking out. Can we get a release if everything is good?


Saving a patch should give a visual indication that it was successful

Currently if you save a patch there is no visual indication that the save succeeded or failed. Pressing 'Save' gives no feedback. Suggest adding a pop up that says if the save succeeded or not.

This will be implemented in next version.

Completed


Create "ClinEpiDB Outreach" user account

@jtlong3rd please create an account that staff can use to create public strategies. We would like the user name to be “ClinEpiDB Staff” and the institution to be "VEuPathDB"
Please hook this new account up so that we can use it to make public analyses in phase 1

can we also update https://wiki.apidb.org/ with this email address & password, & indicate that it should be used when making public/example analyses?


Apply Niche optimization to EspError, add infallible helper function

I'm sure this can optimize the stack for a lot of methods returning Result<(), EspError> out there.
I also added an infallible method using eager const evaluation. There are a lot EspError::from(ESP_ERR_*).unwrap() in esp_idf_hal and esp_idf_svc that could be replaced with EspError::from_infallible::<ESP_ERR_*>().

Should I carve 0.32.1 out of this when it's merged?


fix: update semantic-release dependencies

:tada: This PR is included in version 0.34.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Improve performance of persistent write-ahead log

See suggestions in #56 .
Tasks:

[X] Add SkipAsync to IAsyncBinaryReader
[X] Move identifier of the log record to its metadata.
[ ] Rewrite lock mechanism of PersistentState in a way that allows to do the compaction in parallel with writes or reads
[ ] Bufferize log entries at transport level
[ ] Introduce large-size log entries with externally stored BLOB
[ ] Split file buffer size of regular log entries and snapshot entry
[X] fsync for every write must be disabled by default (FileOptions.WriteThrough flag)

@potrusil-osi, you was right about SkipAsync. The bug in StreamSegment.Position property setter. Now it's fixed. Also, the second minor performance improved is also pushed to develop branch. Command identifier used by Interpreter Framework is now a part of log entry metadata and doesn't require serialization. However, there is another side of this change: log entry representing snapshot must not be passed to the interpreter. Snapshot doesn't have its own identifier.

@sakno, it is unfortunate that I cannot pass snapshot to the interpreter anymore. I currently use it to store the in-memory state and the snapshot command is how it is often initialized. It is easy and elegant. Isn't there a way how to work around the limitation?

Could you please provide a simple example pf how you use custom command type for the snapshot? Do you serialize identifier manually inside of SnapshotBuilder?

@potrusil-osi , support of snapshot entries returned back. CommandHandler attribute has special parameter indicating that the marked method is a handler for snapshot log entry:
[CommandHandler(IsSnapshotHandler = true)]
public ValueTask HandleSnapshot(SnapshotCommand command, CancellationToken token)
{
}

Thanks for such a fast response! I guess you don't want me to give you the example anymore... But no, I was no serializing the identifier inside SnapshotBuilder - it just used its own instance of CommandInterpreter to apply individual entries (including the previous snapshot) and then wrote the SnapshotCommand-based entry at the very end. It worked.
After getting the latest changes (and updating the CommandHandler for snapshot) I'm consistently getting this exception in a follower:
Unhandled exception. DotNext.Net.Cluster.Consensus.Raft.PersistentState+MissingLogEntryException: Log entry with relative index 1 doesn't exist in partition ...\Logs\partition-0\sp-0\0
   at DotNext.Net.Cluster.Consensus.Raft.PersistentState.Partition.ReadAsync(StreamSegment reader, Memory`1 buffer, Int32 index, Boolean refreshStream, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/PersistentState.Partition.cs:line 86
   at DotNext.Net.Cluster.Consensus.Raft.PersistentState.ReadAsync[TResult](LogEntryConsumer`2 reader, DataAccessSession session, Int64 startIndex, Int64 endIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/PersistentState.cs:line 170
   at DotNext.Net.Cluster.Consensus.Raft.PersistentState.ReadAsync[TResult](LogEntryConsumer`2 reader, Int64 startIndex, Int64 endIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/PersistentState.cs:line 234
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.<DotNext.Net.Cluster.Consensus.Raft.IRaftStateMachine.MoveToCandidateState>g__PreVoteAsync|76_0(Int64 currentTerm) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 750
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.DotNext.Net.Cluster.Consensus.Raft.IRaftStateMachine.MoveToCandidateState() in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 717
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__139_1(Object state)
   at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

The 0 partition file mentioned in the error is completely empty (unlike the same file in the leader).

Physical layout of log entries on the disk in 3.1.0 is not compatible with previous versions. As a result, if new PersistentState uses files generated by previous version then you may have unpredictable results. I'll include this fact in release notes. Currently, you need to delete the entire directory with log entries and snapshot.

I delete the logs every time before my tests. I'll keep digging myself what could be causing it...

Fixed. Command id was not passed over the wire correctly.

Everything works well now. Thanks!

@sakno, after all the optimizations I'm getting much better numbers... The suggestion to have a buffer with a different size during compaction (described in #56) still holds. I have modified dotNext locally to give me the option and writing to the snapshot file is much faster thanks to that. I set the buffer to approximately the size of snapshot.

Great news! Currently I'm working on buffered network I/O that allows to copy log entry content passed over the network before calling AppendAsync. This will reduce lock contention time between replication process and user code calling AppendAsync manually. Additionally, your suggestion has been added to the task list.

Did you change the buffer size passed to FileStream constructor? There is may be another issue. I'm using FileOptions.WriteThrough to bypass any intermediate buffers. For me, it's strange that bufferSize has any sense. Could you please remove WriteThrough flag and check performance again? Without this flag, I/O is less durable however it might be more performant. If so, I'll add this flag to configuration of PersistentState.

I'm getting about 50% performance increase without that flag! I wonder if I need to test everything again to figure out the ideal buffer sizes.

I guess without WriteThrough flag the size of the buffer has small impact on performance.

Now WriteThrough flag is disabled by default but can be enabled using Options.WriteThrough property. Changes are pushed to develop branch.

Buffer sizes are now separated in Options class.

@potrusil-osi , I'm finished buffering API for log entries. I'm expect to use in two use cases:

For buffering before calling AppendAsync from user code
For buffering before calling AppendAsync by transport infrastructure

In case of Interpreter Framework, you can use it easily as follows:
var entry = interpreter.CreateLogEntry<MyCommand>(...);
using var buffered = await BufferedRaftLogEntry.CopyAsync(entry, options, token);
log.AppendAsync(buffered, token);

This make sense only for large log entries. Also, you need to experiment with settings from RaftLogEntryBufferingOptions class which instance must be passed to CopyAsync method. It has MemoryThreshold option. If the size of log entry is larger than this threshold then disk persistence will be used for buffering. This is relevant for transport-level buffering because read from the disk is faster than read over the network but probably make no sense for custom log entries acquired constructed by Interpreter. As a result, use buffered log entries for appropriate commands.

Now I'm in a halfway to a new version of PersistentState with parallel compaction. It will have the following characteristics:

Log compaction and appending new entries can be done in parallel
Parallel reads are allowed
Reads are mutually exclusive with log compaction or writes (so you cannot read in parallel with log compaction)

At the moment I implemented one innovative thing: concurrent sorted linked list for partitions. This hand-written data structure is much better than SortedDictionary<K, V> from .NET that has O(log n) in comparison to O(1) average for search provided by a new structure. Moreover, new structure provides lock-free adding/removing operations.
The remaining half way is about writing special lock type on top from AsyncTrigger.

@potrusil-osi , everything is ready for beta-testing. It was really challenging :sunglasses: There are few options for background compaction are available:
Option # 1:
If you're using UsePersistenceEngine extension method for registering custom WAL derived from PersistentState and PersistentState.Options.CompactionMode is set to Background then the library automatically registers background compaction worker.
By default, the worker implements incremental compaction. It means that the only one partition will be compacted at a time. It gives you the best performance with minimal interference with writes. However, this approach has drawbacks: if there are too many writes then disk space will grow fast than compaction.
To replace incremental compaction, you need to implement DotNext.IO.Log.ILogCompactionSupport interface in you custom WAL. It requires only one method. This method will be called by compaction worker on every commit. As a result, you are able to define how deep the compaction should be performed. PersistentState.ForceCompactionAsync(long count, CancellationToken) now gives this opportunity. The first parameter accepts compaction factor. Zero means that no compaction should be performed. 1 means that compaction should be minimal (as in incremental approach). In other words, this factor means how many partitions you want to squash. The number of available partitions for squashing can be requested via PersistentState.CompactionCount property.
Option # 2:
If you're not using UsePersistenceEngine extension method then you need to write Hosted Service and register it in DI container. The service is responsible for running compaction in the background. You can use built-in background compaction worker as an example.
I would appreciate if you will able to contribute a good benchmark. Unfortunately, BenchmarkDotNet is not applicable for load testing. Also, it would be great if will share the final measurements of your application with background compaction.

As an option, EWMA can be used as a base for adaptive algorithm for background compaction. During application lifetime the algorithm can compute exponentially weighted average value of compaction factor and pass it to ForceCompactionAsync method. It's really trivial. However, its application is limited to situation when application has floating workload. If not, EWMA will give the maximum compaction factor which leads to high interference with writes. Moreover, the existing incremental approach should work fine with floating workload.

hey @sakno, thank you for so much work! I'll try to take advantage of the most recent changes as soon as possible.
I have two questions:

What exactly did you mean when you asked me for "a good benchmark"? So far, I'm testing the code as part of a bigger application. It would be possible to extract just the state machine (the "simple queue" I described earlier) and simulate its usage by the application, but then question is if that specific state machine is a good benchmark for you since it is so simple and specific to our use case (it supports just adding items and clearing the queue up to a certain sequence number). Which leads to the next question...
I already took advantage of the first version of ForceCompactionAsync which helped in many areas. But to take it to the next level I'd like to optimize the clear operation so that:

Compaction runs only after the clear is committed - that is already done.
When the compaction runs, it doesn't even try to read those partition files that contain only items have been cleared. This is obviously not possible now. I have optimized this process by skipping the deserialization of most of the content of such items, but the partition files are still processed, metadata read, etc. I'm not sure how exactly to support such a use case. Can you think of anything? Or am I asking for too much here?

Another compaction mode has been added in the latest commit. Now there are three compaction modes:

Sequential which is the only compaction mode available in 3.0.x or earlier. Offers the best optimization of disk space by the cost of performance.
Background which allows to run compaction in parallel with writes
A new one, Foreground which is running during commit process in parallel with applying of the committed entries. It requires no ceremony as Background but much more efficient than Sequential.

Foreground compaction mode tries to squash the same number of previously committed entries in parallel as the number of entries requested for the commit. For instance, you have 3 previously committed entries and you want to commit another 3 entries. In this case, compaction process squashes 3 previously committed entries in parallel with applying new 3 committed entries. If overridden ApplyAsync method has approx the same performance as overridden SnapshotBuilder.ApplyAsync then overhead will be close to zero.

Also, I have an idea about your second question. I can provide optional interface called IRandomAccessSnapshotSupport with a single method. You can implement this interface in the class derived from SnapshotBuilder. In this case I'll able to provide existing snapshot as read-only memory-mapped file and you will get random access to the snapshot contents at high speed instead of sequential reading and skipping. Of course, by the cost of your RAM (but without GC pressure). What do you think?

I think I didn't explain my use case very well. The biggest pain point is that when the compaction is executed on the background there are many partition files full of add operations (the snapshot file is quite small). When clear is committed and the compaction is triggered, almost all of the partitions (including the previous snapshot) can be dropped, except add entries in the last few partitions (which are squashed into a relatively small new snapshot).
To support that specific use case, the state implementation would probably need to be able to set custom metadata for the snapshot and all the partitions, access it during compaction, and influence the compaction algorithm to skip/delete certain partition files based on custom logic. But I'm not sure all this would make any sense for a general purpose WAL.

All this would be nice if the clear operation always cleared everything before. But that is not the case. It has a sequence number parameter which means that all items with sequence numbers less that the provided one should be cleared. Those sequence numbers are provided by the application. That's why I was thinking about custom metadata assigned to snapshot or partitions files - in my case it would be the highest sequence number in that file.

On the other hand, I could easily remember mapping between sequence numbers and entry indexes... In that case the AdjustStartIndex(ref long index) would be really interesting. Especially if you optimized it even more - if the new start index is bigger than the index of the last entry in a file, then the file can be skipped...

Yes, it will allows to skip partition files without reading from them.

SnapshotBuilder.AdjustIndex has been added to develop branch. In your case all skipped partitions remain unread and removed at the final stage of the compaction. But a little performance overhead still exist: partition table is now organized as sorted linked list. If your switch the index in a random way then finding the right partition cannot be done in O(1). Anyway, this overhead is very small in comparison with I/O operations.

@sakno, I'm looking how to create the mapping between sequence numbers and entry indexes and it is currently not that easy. I'm using the interpreter framework which deserializes an entry through the formatter. But because the formatter doesn't have access to the LogEntry object anymore, there is no way to create the <sequence number, entry index> pair (the sequence number is being deserialized). Well, I could work around it by setting the "latest entry index" property on the interpreter (by whoever uses the interpreter, i.e. the state or the snapshot builder), but it is not very elegant for multiple reasons. Could you rather make the current LogEntry available (either in the formatter or the interpreter)?

You can remember this index using ApplyAsync method. Check IsSnapshot property of LogEntry type.

I have the latest version and have seen a new exception when the application was shutting down:
Unhandled exception. System.AggregateException: One or more errors occurred. (Cannot access a disposed object.
Object name: 'AsyncExclusiveLock'.)
 ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'AsyncExclusiveLock'.
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at DotNext.Threading.Tasks.CompletedTask`2.WhenCanceled(Task`1 task) in /_/src/DotNext/Threading/Tasks/CompletedTask.cs:line 24
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location where exception was thrown ---
   at DotNext.Threading.AsyncLock.TryAcquireAsync(TimeSpan timeout, Boolean suppressCancellation, CancellationToken token) in /_/src/DotNext.Threading/Threading/AsyncLock.cs:line 252
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.DotNext.Net.Cluster.Consensus.Raft.IRaftStateMachine.MoveToCandidateState() in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 717
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__139_1(Object state)
   at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

It's an error that exist for a long time since 1.x. Usually, it has no effect on application and you can ignore it.

Hi @sakno, it was a bit tricky to figure out the snapshot index. First, I was struggling where exactly I should be waiting for the snapshot to capture its index. It cannot be PersistentState.ApplyAsync because it is not called with the latest snapshot. It could theoretically be SnapshotBuilder.ApplyAsync, but I should not be capturing the index of a snapshot entry but rather index of the last entry. The problem with this approach is that SnapshotBuilder.ApplyAsync doesn't need to be called at all in case all partitions being compacted are skipped (thanks for the new AdjustIndex). So what I ended up doing was to capture the second argument of AdjustIndex, endIndex. It is a bit tricky, but it works.
I was benchmarking compaction with the latest changes and it was quite slow in my specific case (a lot of partition files, all of them are skipped). In fact, it was so slow (hundreds of milliseconds) that it often caused the leader to lose its leadership because of the compaction lock being held the whole time. Almost all the compaction time was spent in RemovePartitionsAsync. When I locally modified that method to execute the partition disposal on the background (the code after the partitions are detached) everything was way faster. After that change the compaction takes only couple of milliseconds and the leadership is stable. Would it be possible to do something about it officially?

PersistentState.ApplyAsync is called with latest snapshot just once - at application startup (if ReplayOnInitialize is configured). This is when you can remember it. Afterwards, you can catch it in SnapshotBuilder.ApplyAsync and update easily.
I'm curious what exactly slow in RemovePartitionsAsync? There are only two potentially heavy operations:

await partition.DisposeAsync().ConfigureAwait(false)
File.Delete(fileName)

It would be great if you have a chance to measure these two lines of code separately. DisposeAsync should not be slow because all buffers at the time of the call are flushed already. File.Delete probably yes, but usually OS just erases file metadata from the filesystem table without filling with zeroes of its content. Removing partitions out of the lock is very dangerous:

Readers may want to acquire some entries from these partitions
Another compaction may be triggered

What I can do:

Remove partitions in parallel. This will have no effect if your disk drive is a classic HDD and not SSD. Moreover, if it's SSD then controller may not support many parallel writes. So I don't like this idea at all.
Remove partitions from main sorted linked list to another collection and then call DisposeAsync and File.Delete without the lock.

I think it was the DisposeAsync that took the longest. But I will measure it again.
For my local testing I did what you describe as option 2). I understood that modifying the linked list without a lock was dangerous, but I wasn't sure if calling DisposeAsync and File.Delete without the lock was safe or not.

DisposeAsync takes a bit longer than File.Delete but not much. They are both quite expensive.

This is because of tests. In the real world, some of your cluster nodes may be too far behind in terms of WAL state. It will force the leader to read from the beginning of the WAL. And this is the moment when the app can crash due to concurrent read/write from the list or even continue to work with damaged state. I think this is irreproducible in any synthetic unit/integration test as well as many situations in distributed/multi-threading scenarios, such as dead locks.

Anyway, I'm trying to implement option 2: I need to remove the head partitions in the linked list when running within the lock and detach them. Then, I can safely release the lock and continue to do heavy I/O operations with these detached partitions without worrying about potential concurrency.

One more thing: PersistentState.ApplyAsync may receive a snapshot not only at startup. If your node too far behind of the leader then snapshot installation procedure may be triggered by it. In this case, the leader will send the snapshot instead of log entries to that node you'll get the snapshot entry in the method.

@potrusil-osi , I have moved partitions cleanup out of the lock for all compaction modes.

Do you have any additional suggestions? If no, I'll start preparing the release.

Not now. We just moved the testing to a testing cluster which might reveal some more things.

@potrusil-osi , there is some additional performance optimization has added recently: you can enable copy-on-read to reduce lock contention between writes and replication process. Replication requires read lock which cannot co-exist with write/compaction. As a result, the duration of read lock depends on network latency. It can be eliminated if we could have a copy of log entries to be replicated outside of the read lock. Now this behavior can be enabled via CopyOnReadOptions configuration property of PersistentState.Options class and increase throughput of writes by the cost of RAM.

@potrusil-osi , final performance improvements now available. There are many options to achieve the best performance with persistent WAL:

Copy-on-read allows to eliminate lock contention between the replication process and writers of new log entries (see previous comment).
Added AppendAsync(TEntry, bool, CancellationToken) and AppendAndEnsureCommitAsync overloads. bool parameter allows to specify whether the entry should be cached in the memory. If passed as true then you don't need to manually create a buffered copy of the log entry. In this case, the entry will be copied to the memory outside of the lock and then stored in the internal cache as well as to the disk. However, subsequent reads (caused by replication, commit or snapshotting) will be done using the memory without the overhead of disk I/O. This is working only if Options.UseCaching property is enabled. Moreover, you can control the eviction policy of the cache using Options.CacheEvictionPolicy property.

Hi @sakno, I like both optimizations! Hopefully I will have time to take advantage of them before the research project this is part of is finished.

Here is benchmark of various PersistentState caching modes on my NVMe SSD:

Method
Mean
Error
StdDev

ReadLogEntriesWithFullCacheAsync
4.230 us
0.0179 us
0.0139 us

ReadLogEntriesWithMetadataCacheAsync
22.512 us
0.4412 us
0.8605 us

ReadLogEntriesWithoutMetadataCacheAsync
42.370 us
0.8241 us
1.4214 us

You can try this benchmark on your machine. Run DotNext.Benchmarks project with dotnet run -c Bench and select PersistentStateBenchmark.

Mine results are not as good as yours...

Method
Mean
Error
StdDev

ReadLogEntriesWithFullCacheAsync
8.268 us
0.0766 us
0.0716 us

ReadLogEntriesWithMetadataCacheAsync
213.049 us
8.0940 us
23.8652 us

ReadLogEntriesWithoutMetadataCacheAsync
227.656 us
4.4002 us
3.9007 us

The actual result depends on the hardware. In case of ReadLogEntriesWithFullCacheAsync benchmark, the result depends on CPU because log entry fully cached in the memory and disk I/O is not needed. The last two benchmarks have dependency on disk I/O. In my case, it's NVMe SSD which uses PCIe as a bus. In your case it's SSD with SATA interface or even HDD. ReadLogEntriesWithMetadataCacheAsync result looks very suspicious because of high value of standard deviation. It seems like some another task was doing its job in the same time with the benchmark.

@potrusil-osi , do you have estimation of time needed to complete your research work? I'm ready to publish release.

We have two more weeks. After the last two optimizations you provided the local results on my computer are pretty impressive. In our testing cluster the results are not as great, so we are digging deeper what the bottlenecks are.

@sakno, back to your analysis of the benchmark results: I also have a NVMe SSD drive (Samsung PM981 1TB). Any thoughts why my results are so much worse than yours? Are you running the benchmark in Windows?

Yes, I ran these tests on Linux. But I have machine with Windows 10:

Method
Mean
Error
StdDev

ReadLogEntriesWithFullCacheAsync
9.568 us
0.1045 us
0.0872 us

ReadLogEntriesWithMetadataCacheAsync
226.568 us
4.5002 us
3.9893 us

ReadLogEntriesWithoutMetadataCacheAsync
284.843 us
5.6811 us
5.3141 us

The results are close to yours. But machine with Windows 10 has SATA SSD instead of NVMe SSD installed on PCIe. I think your laptop has two drives: system partition running on NVMe and user partition running on SATA. Benchmark creates WAL in temp directory. Usually, this directory is mounted to SSD SATA drive in case of two drives by default.
Also, in your case 23.8652 us is very abnormal StdDev. Try to run this benchmark again without any background tasks.

There is only one drive. But after disabling half the services the results are a bit better:

Method
Mean
Error
StdDev

ReadLogEntriesWithFullCacheAsync
6.396 us
0.0646 us
0.0604 us

ReadLogEntriesWithMetadataCacheAsync
106.715 us
0.7805 us
0.6517 us

ReadLogEntriesWithoutMetadataCacheAsync
140.691 us
0.6169 us
0.5771 us

In benchmark I didn't try to choose optimal configuration parameters. However, I changed default buffer size to 4096 bytes which is equal to filesystem cluster size and now the results much better on SATA SSD but remains the same for NVMe. Pull latest changes from develop branch and try again.

After getting the latest develop:

Method
Mean
Error
StdDev

ReadLogEntriesWithFullCacheAsync
6.208 us
0.0295 us
0.0247 us

ReadLogEntriesWithMetadataCacheAsync
59.048 us
0.2609 us
0.2313 us

ReadLogEntriesWithoutMetadataCacheAsync
89.919 us
0.5084 us
0.4507 us

Thanks for sharing results! Also, I managed to remove a few microseconds by optimizing hot paths in code within UnsafeReadAsync method which is workhorse for all reading operations.

Hi @sakno, I am a colleague of @potrusil-osi who is out of the office this week.  We have deployed our application to a cluster on dedicated hardware to continue testing.  Unfortunately, we are not seeing the same degree of benefit from the optimizations that have been seen when testing locally, as our overall throughput numbers are only about 80% of what was achieved in the local testing.  In particular, we are seeing two issues that we would like to get your thoughts on.

Write Lock Acquisition

We are seeing that the write lock acquisition in AppendCached() is taking significantly more time (~36ms) as compared to the writing of the entry to disk in UnsafeAppend() which takes on average (~1ms).  With the compaction improvements, we are seeing an average compaction time of ~9ms.  Do you have any thoughts that might explain why this is taking so long to acquire the write lock in AppendCached()?

Read Lock Acquisition

In the ReadBuffered() and ReadUnbuffered() methods, we are seeing that a read lock is being acquired, which on average is taking ~4ms.  We have configured the PersitentState.Options with CopyOnReadOptions, which from the previous discussion, indicates that that this operation shouldn't require a read lock at all.  Is it expected that there should still be a lock required for this operation?

Hi @afink-osi ,
How many parallel writes do you have? Are reads/writes intermixed?

We are seeing that the write lock acquisition in AppendCached() is taking significantly more time (~36ms) as compared to the writing of the entry to disk in UnsafeAppend() which takes on average (~1ms).

It depends on numbers of parallel writes. For instance, if you have 36 parallel writes and each write operation takes ~1 ms then the last writer need to wait ~36 ms.

Additionally, there is another project FASTER that offers general-purpose and high-performance WAL. It might be a good candidate for Raft but requires a lot of research.

@afink-osi , @potrusil-osi , let's try again. What I've done: cached log entry will not be persisted until commit. As a result, AppendCachedAsync will be completed synchronously in most cases and lock contention will be low.

Hi @sakno, thanks for the quick response and for the clarification!  Yesterday when running tests, we were seeing replication throughput at ~28% the throughput of our baseline (the application running without replication).  Today, we were able to run some tests using the same storage volume as before but with your latest changes, and the results are very impressive! We are now running at about 65% of the baseline which is much more in line with the local tests that @potrusil-osi was running before and we are seeing minimal time waiting for locks.  We also tested using memory instead of the SSD disk (may be acceptable for our use case) for log storage and pushed this further to ~72% of the baseline.
One more question:  In the new method PersistCachedEntryAsync(), there is a call to WriteAsync() but we do not see any call to a Flush() method.  Are these entries being flushed somewhere else?

I missed call to FlushAsync, my bad. Now it's added. Flushing happens only if partition switched or the end of the log reached for better I/O performance especially if cached log entries are small.

Also I added special TryGetBufferWriter method to IAsyncBinaryWriter interface that allows to enable fast synchronous serialization. When you appending log entries with addToCache parameter the writer will return non-null result. This method is supported only if IAsyncBinaryWriter backed by pipe or dynamic in-memory buffer. In other cases, it will return null.

@potrusil-osi , @afink-osi , I've added support of diagnostic events to persistent WAL. You can configure them via PersistentState.Options type:

LockContentionCounter measures the rate of lock contention
LockDurationCounter measures the duration of the lock in acquired state, in ms
ReadCounter measures the rate of reads, in the number of log entries
WriteCounter measures the rate of writes, in the number of log entries
CompactionCounter measures the rate of log compaction, in the number of log entries
CommitCounter measures the rate of commits, in the number of log entries

You can use dotnet-counters tool from .NET SDK to track the counters or submit values to the monitoring tool.

@potrusil-osi , @afink-osi , I would like to ask you guys to check mmap-wal branch. It contains another optimization of writes based on memory-mapped files. Read performance remains the same. Thanks in advance!

@sakno The new counters look very helpful, I didn't get a chance to try them yet but will hopefully look at them tomorrow.  We were able to test the new changes to develop branch, and we are seeing an additional 11% increase in throughput in our application compared to before these latest optimizations!

Nice to hear that, @afink-osi ! Let me know when you check these counters.

@afink-osi , @potrusil-osi , did you have a chance to finalize your research project? If so, I would like to release new version of .NEXT.


added basic auth

This allows for a basic auth check so that other people can't nefariously get to your cache.

Awesome! Please ping me once it's ready for review.

Hello @buchgr, Thanks for this cool tool.  I'm somewhat new to Bazel, but this auth addition will help my team (I work for Nordstom Technology) get more out of Bazel in our shared pipelines and eventually will allow multiple teams to take advantage of the speed of cached builds without stepping on other teams' toes.
Unfortunately, I've hit a snag.   I was hoping to invoke the cache with a call like build --remote_http_cache=http://user:password@bazel-remote-cache.local:8080 But it appears the Authorization header is stripped out by Bazel.  Do you know of any other method to pass the Authentication?
Thanks again.

@bajacondor on it https://github.com/bazelbuild/bazel/pull/4609 :-)

Wow!  Thank you!  The next step is to add TLS listening to this remote cache server.  If you'd like, I can open another merge request for that and help you out.

@bajacondor that would be awesome! I am currently quite busy with other stuff ... keep your PRs coming 🚀 🚀


fix(no-leaked-conditional-rendering): same test and consequent in conditional expression

Not quite sure if I'm right

I'll merge it after you've done the changes.


Project Idea: Javascript Calendar

Description of the project
I will make a simple calendar using vanilla javascript along with html and css .
Additional context
This project is not already present in the existing projects so, it's something new! Please, Assign that to me under GSSOC.

Already exist


Linux-DS Fixes

Current State:
It crashes because of GModDataPack::GetHashFromString nuking memory for some reason(Allocating more than 2GB,
tcmalloc: large alloc<PHONE_NUMBER> bytes == (nil))

I will implement a detour for the GModDataPack later, since it should be easier to recreate the datapack function than to hope to find the issue.,

Most issues got fixed but It's still broken until a fix for the Datapack is made.


authentication documentation

Is your feature request related to a problem? Please describe.
how can I pass the USER and PASSSWORD arguments that I would pass in curl with the -u option?

You start by reading the documentation.
If you need basic http auth, you can use it like this:
--input=http://name:password@production.es.com:9200/my_index


some functions not worked

ImportedContacts,NOTRETURN_CLIENT,Video Message and some functions not worked some times.
Do you have a road map for project stability?

https://github.com/nebulaim/telegramd/blob/master/doc/road-map.md


Does bracket notation work?

var obj = {
        'bracket:string': 'This string must be accessed with brackets'
    }

obj['bracket:string']
//'This string must be accessed with brackets'

access(obj, "['bracket:string']")
//undefined

It doesn't look like it unless I'm doing something incorrectly here. Any help would be appreciated.
Thanks!

Bracket notation doesn't work, but internally, safe-access uses bracket notation and the strings that you pass in to safely access properties, so you can do this:
var obj = {
  'bracket:string': 'This string must be accessed with brackets'
};

access(obj, 'bracket:string');

For deeply nested objects, you can simply use dot notation:
var obj = {
  'bracket:string': {
    'str@nge-symbols': 'This string must also be accessed with brackets'
  }
};

access(obj, 'bracket:string.str@nge-symbols');

This workaround will not work when a property name will have a '.' in the name. So, it will be useful to have bracket notation.
Thanks!

Agree with @galusben. From server we receive objects like:
{   attributes: {     'product.displayName': 'A product'   } }


Handle expiring/expired auth tokens from github

By default github times out github app user to server access tokens after 8 hours. This is not the case for Oauth apps. Backstage does not currently handle github access tokens that timeout and shows an alert "Bad credentials"
Feature Suggestion
It would be fantastic if the github oauth feature was able to recognise that the token has expired and then renegotiate a new token with github.
Possible Implementation
I havent looked into it in great detail, however the response from the token API contains an expiresAt field. It can be used to determine if a token is no longer valid.
Context
By using a singe github app for frontend (user to server) and backend (server to server) auth, we reduce the complexity of configuration supporting backstage. There is simply one app to create in github that can be used for both. Enabling a security feature such as this (user to server token timeout) is desirable.

Most auth clients handle expiring tokens, and it's done through the RefreshingAuthSessionManager. The GitHub case is a bit tricky, as you have both this issue and the fact that permissions are handled completely differently with GitHub Apps.
To really tidy up things I think we need a bit of a rework of how auth works towards SCMs, but for now I'd be happy with any kind of solution to this tbh :grin:. I wonder if it's for example possible to just use the plain OAuth2 implementation for GitHub apps and registering that as the implementation of the GitHub Auth API instead?


Improve any existing project

Issue Type:

[ ] Bug Report
[x] Feature Request
[ ] Documentaion

Describe the bug

A clear and concise description of what the bug is.

Possible solution

Describe the solution you thought of.

Screenshots

If applicable, add screenshots to help explain your problem.

I want to add Readme file in each of the project. Can I do that?

Yeah sure   @RiyaGupta89  go ahead

Hey, can I add a basic JS project for generating random colors?

sure

Hello, I would like to add a tick sound to the analog clock so as to make it more realistic and at the same time try to incorporate an alarm facility. Also I looked at the bug in the smooth transitioning part in the .js file and would like to figure out a solution for that too. So I want to be assigned to the analog clock project.

ok cool @leader282 go ahead

hey @leader282 if you r gssoc participant the please wait for some time if you make PR before time period starts it will not count.
wait for channels to be made on discord

Okay sure. Thanks for the opportunity sir.

On 24-Feb-2022, at 11:26 PM, Arpit Jain @.***> wrote:
hey @leader282 https://github.com/leader282 if you r gssoc participant the please wait for some time if you make PR before time period starts it will not count.
wait for channels to be made on discord
—
Reply to this email directly, view it on GitHub https://github.com/arpit456jain/Amazing-Js-Projects/issues/62#issuecomment-1050112211, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXNVSWHVUVF2XOTF2AULL5LU4ZWN5ANCNFSM5BGDOUKA.
Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.

I want to work on this issue, can you please assign me?

Respected Sir,
Can we start making pull requests now for GSSOC 2022 since its 1st March and already channels have started forming on discord? Also will any contribution be considered for GSSOC 2022 or we need to contribute particularly in the issues labelled with GSSOC label?
Regards,
Aritra Chakraborty
Contributor
GSSOC 2022

On 24-Feb-2022, at 11:26 PM, Arpit Jain @.***> wrote:
hey @leader282 https://github.com/leader282 if you r gssoc participant the please wait for some time if you make PR before time period starts it will not count.
wait for channels to be made on discord
—
Reply to this email directly, view it on GitHub https://github.com/arpit456jain/Amazing-Js-Projects/issues/62#issuecomment-1050112211, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXNVSWHVUVF2XOTF2AULL5LU4ZWN5ANCNFSM5BGDOUKA.
Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.

Respected Sir, Can we start making pull requests now for GSSOC 2022 since its 1st March and already channels have started forming on discord? Also will any contribution be considered for GSSOC 2022 or we need to contribute particularly in the issues labelled with GSSOC label? Regards, Aritra Chakraborty Contributor GSSOC 2022
…
On 24-Feb-2022, at 11:26 PM, Arpit Jain @.***> wrote: hey @leader282 https://github.com/leader282 if you r gssoc participant the please wait for some time if you make PR before time period starts it will not count. wait for channels to be made on discord — Reply to this email directly, view it on GitHub <#62 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXNVSWHVUVF2XOTF2AULL5LU4ZWN5ANCNFSM5BGDOUKA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.

yes

hey @arpit456jain , i am a GSSOC participant, can I add a Project?

hey @arpit456jain  , I am new to GSSOC  , So I just had a doubt that if we want to contribute or add any project then how can we add and how can we get assigned to a particular project


Add Nexthop selection strategies and @strategy tag to remap.config.

This PR adds NextHop selection strategies that may be used in place of the parent selection strategies in parent.config.  The NextHop selection strategies are used in remap.config with a new @strategy tag.  NextHop strategies are defined in a new config file, 'strategies.yaml'.  When the remap.config is loaded, the NextHopStrategy factory loads the strategies.yaml and creates and configures instances of NextHopSelectionStrategies.  When remap.config is parsed, a strategy is associated with a remap entry if there is a tag '@strategy=$name' in the config.  If the strategy is configured for a remap, wrapper functions used for parent selection in HttpTransact will prefer the NextHopSelection strategies over the ParentSelection strategies when searchin for a NextHop or parent host.   Sample strategy.yaml files are at proxy/http/remap/unit-tests.

This is a big PR. I'll try to read through this over the next week.

@randall the #include is used to concatenate multiple file streams into a single stream as a YAML document that then may be passed.  @amc idea of using 'include: [ file1.yaml, file2.yaml] complicates this IMHO.  I'd have to parse a file, look for a YAML 'include:' name space then re-read all the files so that I can  parse 'strategies'.  I think this makes it too complicated and I think the '#include file' is simpler and more flexible.  I do like @amc's idea of reading a directory and concatenating YAML files in 'ls' order and I'll probably add that.  But, using a YAML list to include other files just seems to complicated to me so, I'm leaving the '#include' for now.

Lots of comments and questions. Only blocking concerns are:

REMAP_OPTFLG_STRATEGY 0x800000000u is too big.
NextHopSelectionStrategy::markNextHopDown isn't called from anywhere.

@a-canary this is fixed.

@a-canary I squashed the 3 commits to one

@a-canary I re-worked NextHopSelectionStrategy::markNextHopDown() where I used std::mutex and std::lock_guard.  Could you take another look at that please?

[approve ci FreeBSD]

@a-canary I re-worked NextHopSelectionStrategy::markNextHopDown() where I used std::mutex and std::lock_guard. Could you take another look at that please?

+1

rebased to pick up fix for FreeBSD CI build

+1

I am still looking into this PR

@gtenev @vmamidi This code isn't perfect. But it's a step in the right direction. I expect this to evolve and mature greatly over the next few releases. We'll start testing this for production in Q1.

+1. I agree I’ve already seen the code. I am now waiting for Gancho’s feedback.

On Nov 12, 2019, at 7:48 AM, a-a-ron<EMAIL_ADDRESS>wrote:

@gtenev @vmamidi This code isn't perfect. But it's a step in the right direction. I expect this to evolve and mature greatly over the next few releases. We'll start testing this for production in Q1.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

rebased to pick up yaml 0.6.3 update.

@calavera still working on some review comments.  It'll go in soon.

This is breaking big time on the CI, segfaulting the tests.


Do not track /Blazor as a single request.

Is your feature request related to a problem?
The GET "/blazor" operation is skewing all time-based results because it looks like a single request (that can be very long).
Describe the solution you'd like.
Track the request as individual operations so that the durations better represents what's actually happening
Describe alternatives you've considered.
Ignore "/blazor" is the only thing I can think of, but I am unsure what else I'll miss.
Additional context.
Blazor server
.NET 6
Hybrid blazor and razor pages
Installed SDK (in startup)
App services on
Linux
Below you can see a 5-minute operation that actually had many super-fast operations.

Based on my experience this still seems to be an issue.


Allow end portal frames and spawners to drop as items using movableEndPortalFrames & movableSpawners respectively

Following #52, it was found that the blocks would not drop as items when pushed by a piston and tnt explodes as soon as they're pushed.
This improvement/feature would allow for them to drop as items when pushed by a piston. Likely will require a mixin targeting the PistonBlockEntity class.
This should not be its own separate rules but instead to extend the existing "movable" rules to have a third allow value. The allowed values should be:

true
false (default)
dropAsItemOnExplosion

This won't be included in v1.0.0 but a future version. Likely v1.1.0.

Injection method should be PistonExtensionBlock.getDroppedStacks with a target for pistonBlockEntity.getPushedBlock().getDroppedStacks(builder); This is where a pistonBlockEntity has dropped stacks from.

This would be a minor version update. Old value for the rule is parsed correctly so it is not a breaking change.

Decided this can be included in v1.0.0 but with end portal frames only. Spawners is kind of pointless as no nbt data can be saved.

Spawners can be revisited in the future, have a potential idea but not too bothered for v1.0.0


Secondary Indexes Pt. 1

So what's here?

Creating, renaming, and dropping indexes work through all SQL paths EXCEPT through CREATE TABLE
The secondary indexes supported are only the non-unique, non-spatial, and non-fulltext variant. This includes both non-nullable and nullable columns.
Everything persists to disk
INSERT, UPDATE, and DELETE statements all update any indexes that are attached to the table.

What's not here?

Select doesn't make use of the secondary indexes whatsoever
Index tables show up in dolt status and everywhere else you can think of
SHOW INDEX isn't implemented
Modifying a table's schema doesn't update the indexes
Deleting a table doesn't delete the attached indexes
Truncating a table doesn't truncate the indexes
Committing a table doesn't automatically commit its indexes
Unique hasn't been added as a constraint

This is going to fail Jenkins as I'm just referencing my local copy of go-mysql-server for ease of development, however rest assured that I ran both the bats tests and go tests locally and everything passed, so I haven't messed any of that stuff up (that we have covered at least).

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

What's still missing?

Tests. All of the tests. I'm 99.9% sure that everything is working perfectly, as I'm still working on the tests. I've already caught some stuff, so instead of including some of the tests here, this is everything sans index-specific testing.
Unique indexes don't yet enforce anything. It required working indexes first, so while I attempted it earlier, I decided to get the partial keys in first instead of making another hack, just to get replaced anyway.
SHOW INDEX doesn't show anything. I don't think it's critically important, so I'll make another PR in the future to add it.
dolt diff doesn't display index additions or modifications. Fairly critical functionality, so it's next on the menu, but that should also be a separate PR in my opinion. This one is already too large.
Speed is pretty bad. With the missing partial key support, they're not performant.

So, my priority list on the missing things are as follows:

Tests
Diffs
Partial Key Support
Unique Constraint

Besides what's missing, go ahead and give this the full review treatment. It's fully passing all existing tests, so any additional non-feedback changes to the code will come from bugs discovered in testing, which will be a separate PR.

PR feedback, test fixes, and the bats tests now have their own commits under the Commits tab, so if you're interested in those, you can check them out individually there.


LinkPopover achorEl issue on multiple RTE

Checklist

[x] This is not covered in the repository examples.
[x] I am using the package latest version.
[x] This is a bug report or an enhancement proposal.

Description
When rendering multiple RTEs in one screen. The linkPopover is always anchored to the very first rendered RTE link-control button.

The same happened to me

I just came across this same thing. You need to pass a unique id to the MUIRichTextEditor component. This will anchor the popover in the correct spot.


Make contact page

Tested functionality, works for me 👍🏽

Yes, it works, I just need to add some edits as well here for the event default images. thanks


[Event Request] Report 99001015 - OnBeforeWithReqLine

Please add a new integration event OnBeforeWithReqLine in Report 99001015.
    [Integration]
    LOCAL PROCEDURE OnBeforeWithReqLine@1000(VAR ProdOrderRoutingLine@1001: Record 5409;VAR WorkCenter@1002: Record 99000754;VAR RequisitionLine@1003: Record 246;VAR IsHandled@1004: Boolean);
    BEGIN
    END;

Call for function:
    LOCAL PROCEDURE InsertReqWkshLine@3();
    VAR
      IsHandled@1000 : Boolean;
    BEGIN
      ProdOrderLine.CALCFIELDS("Total Exp. Oper. Output (Qty.)");

+     IsHandled := FALSE;
+     OnBeforeWithReqLine("Prod. Order Routing Line","Work Center",ReqLine,IsHandled);
+     IF IsHandled THEN
+       EXIT;

      WITH ReqLine DO BEGIN
        SetSubcontracting(TRUE);
        BlockDynamicTracking(TRUE);

        INIT;
        "Line No." := "Line No." + 10000;
        VALIDATE(Type,Type::Item);
        VALIDATE("No.",ProdOrderLine."Item No.");
        VALIDATE("Variant Code",ProdOrderLine."Variant Code");
        VALIDATE("Unit of Measure Code",ProdOrderLine."Unit of Measure Code");
        VALIDATE(Quantity,QtyToPurch);
        GetGLSetup;
        IsHandled := FALSE;
        OnBeforeValidateUnitCost(ReqLine,"Work Center",IsHandled);
        IF NOT IsHandled THEN
          IF Quantity <> 0 THEN BEGIN
            IF "Work Center"."Unit Cost Calculation" = "Work Center"."Unit Cost Calculation"::Units THEN
              VALIDATE(
                "Direct Unit Cost",
                ROUND(
                  "Prod. Order Routing Line"."Direct Unit Cost" *
                  ProdOrderLine."Qty. per Unit of Measure",
                  GLSetup."Unit-Amount Rounding Precision"))
            ELSE
              VALIDATE(
                "Direct Unit Cost",
                ROUND(
                  ("Prod. Order Routing Line"."Expected Operation Cost Amt." -
                   "Prod. Order Routing Line"."Expected Capacity Ovhd. Cost") /
                  ProdOrderLine."Total Exp. Oper. Output (Qty.)",
                  GLSetup."Unit-Amount Rounding Precision"));
          END ELSE
            VALIDATE("Direct Unit Cost",0);
        "Qty. per Unit of Measure" := 0;
        "Quantity (Base)" := 0;
        "Prod. Order No." := ProdOrderLine."Prod. Order No.";
        "Prod. Order Line No." := ProdOrderLine."Line No.";
        "Due Date" := "Prod. Order Routing Line"."Ending Date";
        "Requester ID" := USERID;
        "Location Code" := ProdOrderLine."Location Code";
        "Bin Code" := ProdOrderLine."Bin Code";
        "Routing Reference No." := "Prod. Order Routing Line"."Routing Reference No.";
        "Routing No." := "Prod. Order Routing Line"."Routing No.";
        "Operation No." := "Prod. Order Routing Line"."Operation No.";
        "Work Center No." := "Prod. Order Routing Line"."Work Center No.";
        VALIDATE("Vendor No.","Work Center"."Subcontractor No.");
        Description := "Prod. Order Routing Line".Description;
        OnAfterTransferProdOrderRoutingLine(ReqLine,"Prod. Order Routing Line");

        // If purchase order already exist we will change this if possible
        PurchLine.RESET;
        PurchLine.SETCURRENTKEY("Document Type",Type,"Prod. Order No.","Prod. Order Line No.","Routing No.","Operation No.");
        PurchLine.SETRANGE("Document Type",PurchLine."Document Type"::Order);
        PurchLine.SETRANGE(Type,PurchLine.Type::Item);
        PurchLine.SETRANGE("Prod. Order No.",ProdOrderLine."Prod. Order No.");
        PurchLine.SETRANGE("Prod. Order Line No.",ProdOrderLine."Line No.");
        PurchLine.SETRANGE("Routing No.","Prod. Order Routing Line"."Routing No.");
        PurchLine.SETRANGE("Operation No.","Prod.  & Chg. Qty.";
          "Accept Action Message" := TRUE;
        END ELSE BEGIN
          "Ref. Order No." := ProdOrderLine."Prod. Order No.";
          "Ref. Order Type" := "Ref. Order Type"::"Prod. Order";
          "Ref. Order Status" := ProdOrderLine.Status;
          "Ref. Line No." := ProdOrderLine."Line No.";
          "Action Message" := "Action Message"::New;
          "Accept Action Message" := TRUE;
        END;

        IF "Ref. Order No." <> '' THEN
          GetDimFromRefOrderLine(TRUE);

        OnBeforeReqWkshLineInsert(ReqLine,ProdOrderLine);
        INSERT;
      END;
    END;

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


chore: update tremor and tailwindcss

Description
Hi, Severin from https://github.com/tremorlabs here. I updated the Tremor library as well as tailwindcss and headlessui to their newest versions. Since the Tremor update to v3 has some breaking changes I had to refactor a few files (i.e. tailwind.config and Datepicker).
Two things that I saw using dummy data:

OS systems are mentioned for every name + value combination, shouldn't this be summed up?

Function error does not display anything

Fixes # (issue)
Type of change

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation

How Has This Been Tested?
All default tests are passing
Checklist:

[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have update the CHANGELOG
[x] New and existing unit tests pass locally with my changes

Thanks @severinlandolt! And good catch on those 2 issues, we'll fix those separately


Cannot import a bacpac into an existing non-empty db. Need to clarify that.

Change "Imports a bacpac into an existing database" to  "Imports a bacpac into a new database, or an existing empty database."
Reason: You cannot import a bacpac into an existing non-blank database. The above description has confused customers and caused several support incidents to be open.
If you attempt to import a bacpac to an existing -non empty- database you get this error:
Operation failed with status: 'Bad Request'. Details: There was an error that occurred during this operation : 'Error encountered during the service operation. ; Exception Microsoft.SqlServer.Management.Dac.Services.ServiceException:Target database is not empty. The import operation can only be performed on an empty database.; '

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c1bdd674-b454-e863-c943-9c979b466a92
Version Independent ID: b15341f9-3f1b-00e0-a1c2-970f25a6ab03
Content: az sql db
Content Source: src/azure-cli/azure/cli/command_modules/sql/_help.py
Service: sql-database
GitHub Login: @rloutlaw
Microsoft Alias: routlaw

sql


fix: Use at least one songs in setlists

Description
In a past refactor, we enabled the default minimum songs for a valid setlist to be. As a consequence, we were always using the first setlist found, even if it was empty.
These changes consider a valid setlist to at least have one song and also fix the main backend script to actually use the provided minimum amount of songs.

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


"Server Too Busy" or "Server Unavailable"?

Category

[x] Question
[ ] Typo
[ ] Bug
[ ] Additional article idea

Does the 503 throttling error come with "Server Too Busy" or "Server Unavailable"? We've observed some "Server Unavailable" exceptions when the tenant gets busy, but the doc says" Server Too Busy".
The provided sample code also uses "Server Unavailable".

503 != throttling... 429=throttling... that's different.

The Document says

For all other requests, including CSOM or REST calls, SharePoint Online returns HTTP status code 429 ("Too many requests") or 503 ("Server Too Busy") and the requests will fail.

Do you mean 429 must be throttling. For 503, it only means throttling when Server Too Busy, then Server Unavailable is a different exception?

HTTP status 429 is the response you get when SharePoint reports that your request is being throttled.
That is not the same as a 500-series error which is a server error. Without more context (SharePoint environment, what you're doing to trigger that, etc), can't comment further.


[OSD-10627] handle the ocm response

What type of PR is this?
feature
What this PR does / why we need it?
Handle the ocm response and return corresponding errors
Which Jira/Github issue(s) this PR fixes?
Fixes #OSD-10627
Special notes for your reviewer:
Pre-checks (if applicable):

[ ] Tested latest changes against a cluster
[ ] Ran make generate command locally to validate code changes
[ ] Included documentation changes with PR

Codecov Report

Merging #32 (afdf107) into master (13db761) will decrease coverage by 1.35%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master      #32      +/-   ##
==========================================
- Coverage   35.23%   33.87%   -1.36%
==========================================
  Files          13       13
  Lines         650      676      +26
==========================================
  Hits          229      229
- Misses        410      436      +26
  Partials       11       11

Impacted Files
Coverage Δ

pkg/handlers/webhookreceiver.go
57.29% <0.00%> (-8.98%)
:arrow_down:

The testing is hampered by difficulty in being able to test different forms of Response, as the fields aren't publically exported.
I'll try and address that and the Status issue in a separate PR.
Nothing else seems like a showstopper from me
/lgtm


Update LAB_02L04_Deploy_and_manage_host_pools_and_hosts_with_PowerShe…

…ll_ADDS.md
Module: 00
Lab/Demo: 00
Fixes # .
Changes proposed in this pull request:

Submitted by davemct on behalf of MOC Labs Review


#29 Fixed Media Plugin issue

Fixed the bug: Cannot find name 'MediaPlugin'

ignore this pull request


Port OpenResty's ngx.re.* to envoy Lua filter

Title: Port OpenResty's ngx.re.* to envoy Lua filter
Description:
Background: OpenResty provide a group of regex API called ngx.re. These APIs contain:

ngx.re.gsub: replace all occur value matched the pattern
ngx.re.sub: like ngx.re.gsub, but only called once
ngx.re.match: capture the part of value which matched the pattern
ngx.re.find: like ngx.re.match, but return the index instead
ngx.re.gmatch: create an iterator which match a new value in the each call
ngx.re.split: split the string via regex pattern
and some others.

It would be great if envoy can port these APIs to Lua filter. Of course, we can't be 100% compatible with OpenResty, since OpenResty uses PCRE and Envoy uses RE2/std::regex.
Since CompiledMatcher already exposes the replaceAll, I think we can start this port with ngx.re.gsub. We can use RE2 as regex engine, and implement a envoy.re.gsub with replaceAll. The first step can be a POC.
What about your opinion?

Any idea about it from the Envoy maintainers?

Any idea about it from the Envoy maintainers?


Asking for the numerical dataset of FB15k-num, FB15k-237-num (KBLRN paper)

Could you please share the FB15k-num, FB15k-237-num used in the paper of "KBLRN: End-to-End Learning of Knowledge Base Representation with Latent, Relational, and Numerical Features"?
I tried to reproduce the experiment result of Table 4 in this paper, but I can not create the same set of valid and testing data as Table 1. (".. where numerical features are never used for the triples.")
Thank you very much and I am looking forward to hearing from you.
Best regards,
Phuc

Regarding the numerical data, the stats mentioned in the paper is "This resulted in 116 different numerical features and 12,826 entities".
It seems that the [File 1] in your GitHub repo only has 12,493 entities. Is it the [File 1] was used in this paper?
[File 1]: https://github.com/nle-ml/mmkb/blob/master/FB15K/FB15K_NumericalTriples.txt

Hi Phuc,
i) you are right. Actually there are numerical features for 12,493 entities, and not 12,826. I had a dictionary of that size, but only 12,493 entities had values for this type of features.
ii) The only difference of FB15k-num and FB15k-237-num with respect to the normal versions is the validation and test set. There, we only perform link prediction in those triples where both head and tail have numerical attributes. We wanted to isolate the effect of the numerical expert, and thus we wanted to perform link prediction in that subset. However, the training set is the standard one.
Alberto

Thank you for your respond.
Regarding ii), could you share the subset of validation and test set of FB15k-num, and FB15k-237-num? I mean the validation set and testing set.
FB15k: Valid: 5156 triples, Test: 6012 triples
FB15k-237-num: Valid: 1058 triples, Test: 1215 triples
I did remove those triples as you mentioned on the validation and testing of FB15k-237, and FB15k, but I seem that I got a different list of triples.
The following is the FB15k-237-num triples:
What I got: Valid: 9384 triples, Test: 10934 triples

Hi,
note that to keep a triple both head and tail need to have at least one numerical feature in common.
Please find attached the test set of FB15k-num and FB15k-237-num. Sorry but I dont find the validation files right now... I moved outside NEC and it is hard now for me to locate these files.
Alberto
NF_15ktest_triples.txt
NF_15k237test_triples.txt

Hi Alberto,
It is great. (Test file is OK for me to run the verify experiments)
Thank you very much for your time and effort.
Phuc


alps: 2021-09-29 -> 2021-03-01

Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

What is the rationale for downgrading the package?

My apologies. It's supposed to be 2022-03-01, not 2021-03-01. I will fix that later today, hopefully.

Fixed.


Query API using only short name and version

First, thank you for open sourcing this amazing library. It really is fantastic as it can be expanded not just for MODIS but a ton of other EO datasets!
Is your feature request related to a problem? Please describe.
I see that the API currently uses doi as the parameter. This limits the ability to query/ fetch other datasets. Eg:
We are currently able to fetch only MODIS data since the doi is hard coded
The following is the end point the API is currently fetching
https://cmr.earthdata.nasa.gov/search/collections.json?doi=10.5067/MODIS/MOD13A1.006
But instead we could query the API only using short_name and version as shown below
https://cmr.earthdata.nasa.gov/search/collections.json?short_name=MOD13A1&version=006

Describe the solution you'd like
If we can exclude doi as parameter and instead use short_name and version it would help fetching other datasets as well.
For eg, the following two dataset can be queried in addition to current MODIS implementation.

VIIRS/NPP Leaf Area Index/FPAR 8-Day L4 Global 500m SIN Grid V001 Provider: LPDAAC_ECS
SMAP Enhanced L3 Radiometer Global Daily 9 km EASE-Grid Soil Moisture V004 Provider: NSIDC_ECS

Eg API end point:
https://cmr.earthdata.nasa.gov/search/collections.json?short_name=VNP15A2H&version=001
The current implementation would require us to change the doi to 10.5067/VIIRS for it make a request to the above end point.
Additional context
The current implementation only supports hdf file format, which can be extended to h5, nc and various popular file format.
Would appreciate your thoughts if there is any specific reason for having doi parameter for the end point.

Hi @amanbagrecha, thanks so much for this comprehensive review. You make wonderful points here about expanding the generalizability of MODIS tools. To answer your last question first, I don’t think there’s a super compelling reason not to softcode the doi. We would want to keep backwards compatibility in mind when looking into possible code changes.
As for next steps, I absolutely invite you to fork the repo and draft up a PR if you would like. Otherwise we can include this Open Issue in our backlog which we would triage in the coming weeks/months.

Thanks @jamie-sgro for your input! I'd try to draft a PR within next couple of days and we can discuss any modification on the PR board.


new module: logos

Stuff to cater for logo marks (images) and logo text, something akin to what we have in the design guide inner header.

do you intend to have logos as resource downloads or supply them as CDN paths?

As far as I know we have no CDN so we will give you the assets with npm. I will try to keep pixel graphics out of it as much as I can. So far I know only of one. The rest will be SVGs

In my mind I see this module more as providing support for custom logos, than logos themselves — @elisechant @dominikwilkowski or am I off the mark here?

@klepas no I think you are probably right as well. I suppose that some logos will be consistent like the Commonwealth of Aus Logo, but you're right maybe those sort of resources should be downloaded elsewhere.

I also like the idea of just creating a module for SVGs in general that gives you a place to put them and a way to use them. Pixel graphics should be downloaded as zips (to prevent hot linking) from the site.

@TrebBrennan has mentioned work done on improving the Commonwealth logo SVG - definitely some value in providing canonical versions of this. Some work has also been done on small screen readability.


Align DOI UI and Snapshot UI Flow

┆Issue is synchronized with this Jira Story
┆Issue Type: Story
┆Fix Versions: Dockstore 1.7
┆Sprint: Seabright Sprint 17 Raft
┆Issue Number: DOCK-865
┆Epic: Version Immutability

➤ David Steinberg commented:
Edit dialog disable controls for frozen workflow
Guard against issuing DOI for mutable version

➤ David Steinberg commented:
Include version information in snapshot dialog

Duplicates #2846